Entropy

Entropy describes how unpredictable information is.

$S$ is the sample of training data $p_\oplus$ is the fraction of positive samples $p_\ominus$ is the fraction of negative samples $Entropy(S) \equiv -p_\oplus log_2 p_\oplus - p_\ominus log_2 p_\ominus$

Entropy

$Entropy(S)$ is the expected number of bits needed to encode a class of a randomly drawn member of the sample set (under the optimal, shortest-length code)

Entropy

An optimal length code uses $-log_2 p$ bits to encode a message of probability p

i.e. if $p=0.01$, then the optimal code length is $6.64$

and if $p=0.99$, then the optimal code length is $0.01$

Entropy

Recall that our sample set $S$ has $p_\oplus$ and $p_\ominus$ proportions of $\oplus$ and $\ominus$ classes

Hence, to encode a random member of $S$, weigh the probability of drawing a member of the class by the optimal length code of the class:

$p_\oplus (-log_2 p_\oplus) + p_\ominus ( -log_2 p_\ominus )$

$Entropy(S) \equiv -p_\oplus log_2 p_\oplus - p_\ominus log_2 p_\ominus$

Building a Tree Example

Day Outlook Temp Humidity Windy Play D1 Sunny Hot High False No D2 Sunny Hot High True No D3 Overcast Hot High False Yes D4 Rainy Mild High False Yes D5 Rainy Cool Normal False Yes D6 Rainy Cool Normal True No D7 Overcast Cool Normal True Yes D8 Sunny Mild High False No D9 Sunny Cool Normal False Yes D10 Rainy Mild Normal False Yes D11 Sunny Mild Normal True Yes D12 Overcast Mild High True Yes D13 Overcast Hot Normal False Yes D14 Rainy Mild High True No

Building a Tree Example

Day Outlook Temp Humidity Windy Play D1 Sunny Hot High False No D2 Sunny Hot High True No D3 Overcast Hot High False Yes D4 Rainy Mild High False Yes D5 Rainy Cool Normal False Yes D6 Rainy Cool Normal True No D7 Overcast Cool Normal True Yes D8 Sunny Mild High False No D9 Sunny Cool Normal False Yes D10 Rainy Mild Normal False Yes D11 Sunny Mild Normal True Yes D12 Overcast Mild High True Yes D13 Overcast Hot Normal False Yes D14 Rainy Mild High True No

Searching Hypothesis Space with ID3

Properties of ID3

For any given training set

Is it always possible to build a tree?

If so, will it be a good tree?

Properties of ID3

Only 1 hypothesis (decision tree)

If an early split goes wrong, we're stuck with it!

Uses class statistics, robust to noise

Inductive bias of ID3: tends to prefer shorter trees

Inductive Bias of ID3

Favors shorter trees/more information gain closer to the root

The bias arises from the search, not the search space

Occam's razor: the simplest hypothesis that fits is the best one

Overfitting

Let's describe the error of a hypothesis $h$ as $error(h)$

Now consider specific error measurements:

• training set, $S$: $error_S(h)$
• full dataset, $D$: $error_D(h)$

Overfitting

Hypothesis $h$ overfits if:

$error_S(h) < error_S(h')$

and

$error_D(h) > error_D(h')$

Overfitting

Causes of overfitting

Not enough examples on some attributes

Noisy data

Preventing Overfitting

Stop growing when a new split isn't statistically significant

Grow and prune post-hoc

Reduced-error pruning

• Build a tree as usual, potentially overfitting
• Use a validation dataset
• Greedily remove nodes that improve the accuracy on the validation data

Limitations?

Reduced-error pruning