Information Gain

Information gain measures how much a variable reduces uncertainty about the target variable. For this reason, it is widely used in decision trees to choose which feature should be selected for a split. It is computed through differences in entropy and indicates which feature separates the data most meaningfully. It is also valuable in feature selection and explanatory analysis. Information gain is the information-theoretic answer to the question: “Which variable actually carries useful information?”