Data Science
Entropy
Information theory
Information theory provides a mathematical basis for measuring the information content.
To understand the notion of information, think about it as providing the answer to a question, for example, whether a coin will come up heads.
- If one already has a good guess about the answer, then the actual answer is less informative.
- If one already knows that the coin
is rigged so that it will come with heads with probability
0.99, then a message (advanced information) about the actual
outcome of a flip is worth less than it would be for a
honest coin (50-50).
More details
Entropy
Pr(cj) is the probability of class cj in data set D
We use entropy as a measure of impurity or disorder of data
set D.
Information gain
Information gain is the reduction in
entropy or surprise by transforming a dataset. Information gain
is calculated by comparing the entropy of the dataset before and
after a transformation.
Detailed example