Classification Basic Concepts, Decision Trees, and Model Evaluation
Classification Basic Concepts, Decision Trees, and Model Evaluation
3 No Small 70K No
6 No Medium 60K No
Training Set
Apply
Tid Attrib1 Attrib2 Attrib3 Class Model
11 No Small 55K ?
15 No Large 67K ?
10
Test Set
Two classes
– Predicting tumor cells as benign or malignant
– Classifying credit card transactions
as legitimate or fraudulent
Multiple classes
– Classifying secondary structures of
protein as alpha-helix, beta-sheet,
or random coil
– Categorizing news stories as finance,
weather, entertainment, sports, etc
Decision trees
Rule-based methods
Logistic regression
Discriminant analysis
k-Nearest neighbor (instance-based learning)
Naïve Bayes
Neural networks
Support vector machines
Bayesian belief networks
splitting nodes
Tid Refund Marital Taxable
Status Income Cheat
classification nodes
MarSt Single,
Married Divorced
Tid Refund Marital Taxable
Status Income Cheat
NO Refund
1 Yes Single 125K No
Yes No
2 No Married 100K No
3 No Single 70K No NO TaxInc
4 Yes Married 120K No < 80K > 80K
5 No Divorced 95K Yes
NO YES
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No There can be more than one tree
10 No Single 90K Yes that fits the same data!
10
6 No Medium 60K No
Training Set
Apply Decision
Tid Attrib1 Attrib2 Attrib3 Class
Model Tree
11 No Small 55K ?
15 No Large 67K ?
10
Test Set
Test data
Start from the root of tree. Refund Marital Taxable
Status Income Cheat
No Married 80K ?
Refund 10
Yes No
NO MarSt
Single, Divorced Married
TaxInc NO
< 80K > 80K
NO YES
Test data
Refund Marital Taxable
Status Income Cheat
No Married 80K ?
Refund 10
Yes No
NO MarSt
Single, Divorced Married
TaxInc NO
< 80K > 80K
NO YES
Test data
Refund Marital Taxable
Status Income Cheat
No Married 80K ?
Refund 10
Yes No
NO MarSt
Single, Divorced Married
TaxInc NO
< 80K > 80K
NO YES
Test data
Refund Marital Taxable
Status Income Cheat
No Married 80K ?
Refund 10
Yes No
NO MarSt
Single, Divorced Married
TaxInc NO
< 80K > 80K
NO YES
Test data
Refund Marital Taxable
Status Income Cheat
No Married 80K ?
Refund 10
Yes No
NO MarSt
Single, Divorced Married
TaxInc NO
< 80K > 80K
NO YES
Test data
Refund Marital Taxable
Status Income Cheat
No Married 80K ?
Refund 10
Yes No
NO MarSt
Single, Divorced Married Assign Cheat to “No”
TaxInc NO
< 80K > 80K
NO YES
6 No Medium 60K No
Training Set
Apply Decision
Tid Attrib1 Attrib2 Attrib3 Class
Model Tree
11 No Small 55K ?
15 No Large 67K ?
10
Test Set
Many algorithms:
– Hunt’s algorithm (one of the earliest)
– CART
– ID3, C4.5
– SLIQ, SPRINT
Refund Refund
Don’t
Yes No Yes No
Cheat
Don’t Don’t Don’t Marital
Cheat Cheat Cheat Status
Single,
Tid Refund Marital Taxable Married
Status Income Cheat
Divorced
1 Yes Single 125K No Refund Don’t
Cheat
2 No Married 100K No Yes No Cheat
3 No Single 70K No
4 Yes Married 120K No Don’t Marital
5 No Divorced 95K Yes Cheat Status
6 No Married 60K No Single,
7 Yes Divorced 220K No
Married
Divorced
8 No Single 85K Yes
9 No Married 75K No Taxable Don’t
Cheat
10
10 No Single 90K Yes Income
< 80K >= 80K
Don’t Cheat
Cheat
Greedy strategy
– Split the records at each node based on an
attribute test that optimizes some chosen
criterion.
Issues
– Determine how to split the records
How to specify structure of split?
What is best attribute / attribute value for splitting?
Greedy strategy
– Split the records at each node based on an
attribute test that optimizes some chosen
criterion.
Issues
– Determine how to split the records
How to specify structure of split?
What is best attribute / attribute value for splitting?
Size
{Small,
What about this split? Large} {Medium}
Taxable Taxable
Income Income?
> 80K?
< 10K > 80K
Yes No
Greedy strategy
– Split the records at each node based on an
attribute test that optimizes some chosen
criterion.
Issues
– Determine how to split the records
How to specify structure of split?
What is best attribute / attribute value for splitting?
Greedy approach:
Nodes with homogeneous class distribution
are preferred.
Need a measure of node impurity:
C0: 5 C0: 9
C1: 5 C1: 1
Non-homogeneous, Homogeneous,
high degree of impurity low degree of impurity
Gini index
Entropy
Misclassification error
Attribute A? Attribute B?
Yes No Yes No
M1 M2 M3 M4
M12 M34
Gain = M0 – M12 vs. M0 – M34
Choose attribute that maximizes gain
Jeff Howbert Introduction to Machine Learning Winter 2012 ‹#›
Measure of impurity: Gini index
GINI (t ) 1 [ p( j | t )]2
j
GINI (t ) 1 [ p( j | t )] 2
C1 0 p( C1 ) = 0 / 6 = 0 p( C2 ) = 6 / 6 = 1
C2 6 Gini = 1 – p( C1 )2 – p( C2 )2 = 1 – 0 – 1 = 0
C1 1 p( C1 ) = 1 / 6 p( C2 ) = 5 / 6
C2 5 Gini = 1 – ( 1 / 6 )2 – ( 5 / 6 )2 = 0.278
C1 2 p( C1 ) = 2 / 6 p( C2 ) = 4 / 6
C2 4 Gini = 1 – ( 2 / 6 )2 – ( 4 / 6 )2 = 0.444
k
ni
GINI split GINI (i )
i 1 n
No 0 7 1 6 2 5 3 4 3 4 3 4 3 4 4 3 5 2 6 1 7 0
Gini 0.420 0.400 0.375 0.343 0.417 0.400 0.300 0.343 0.375 0.400 0.420
Greedy strategy
– Split the records at each node based on an
attribute test that optimizes some chosen
criterion.
Issues
– Determine how to split the records
How to specify structure of split?
What is best attribute / attribute value for splitting?
0.9
0.8
x < 0.43?
0.7
Yes No
0.6
y
0.3
Yes No Yes No
0.2
:4 :0 :0 :4
0.1 :0 :4 :3 :0
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
x
Border between two neighboring regions of different classes is known
as decision boundary.
In decision trees, decision boundary segments are always parallel to
attribute axes, because test condition involves one attribute at a time.
Jeff Howbert Introduction to Machine Learning Winter 2012 ‹#›
Classification with decision trees
Advantages:
– Inexpensive to construct
– Extremely fast at classifying unknown records
– Easy to interpret for small-sized trees
– Accuracy comparable to other classification
techniques for many simple data sets
Disadvantages:
– Easy to overfit
– Decision boundary restricted to being parallel
to attribute axes
Jeff Howbert Introduction to Machine Learning Winter 2012 ‹#›
MATLAB interlude
matlab_demo_04.m
Part A
Generalization
Overfitting, underfitting
Validation
Accuracy
a = number of test samples with label correctly predicted
b = number of test samples with label incorrectly predicted
a
accuracy
ab
example
75 samples in test set
accuracy = 62 / 75 = 0.827
Limitations of accuracy
– Consider a two-class problem
number of class 1 test samples = 9990
number of class 2 test samples = 10
– What if model predicts everything to be class 1?
accuracy is extremely high: 9990 / 10000 = 99.9 %
but model will never correctly predict any sample in class 2
in this case accuracy is misleading and does not give a good
picture of model quality
Confusion matrix
example
(continued from actual class
two slides back)
class 1 class 2
class 1 21 6
predicted
class
class 2 7 41
21 41 62
accuracy
21 6 7 41 75
Confusion matrix
actual class
– derived metrics
(for two classes) class 1 class 2
(negative) (positive)
class 1
21 (TN) 6 (FN)
predicted (negative)
class class 2
7 (FP) 41 (TP)
(positive)
Confusion matrix
actual class
– derived metrics
(for two classes) class 1 class 2
(negative) (positive)
class 1
21 (TN) 6 (FN)
predicted (negative)
class class 2
7 (FP) 41 (TP)
(positive)
TP TN
sensitivit y specificit y
TP FN TN FP
matlab_demo_04.m
Part B
– stage of optimization
example: number of iterations in a gradient
descent optimization
underfitting overfitting
optimal fit
Post-pruning
– Grow full decision tree
– Trim nodes of full tree in a bottom-up fashion
– If generalization error improves after trimming, replace
sub-tree by a leaf node.
– Class label of leaf node is determined from majority
class of instances in the sub-tree
– Can use various measures of generalization error for
post-pruning (see textbook)
A1 A4
A2 A3
matlab_demo_04.m
Part C
TRAIN
on
TEST
Training set:
– Used to drive model building and parameter optimization
Validation set
– Used to gauge status of generalization error
– Results can be used to guide decisions during training process
typically used mostly to optimize small number of high-level meta
parameters, e.g. regularization constants; number of gradient descent
iterations
Test set
– Used only for final assessment of model quality, after training +
validation completely finished
Holdout
Cross-validation
Leave-one-out (LOO)
Holdout
– Pro: results in single model that can be used directly
in production
– Con: can be wasteful of data
– Con: a single static holdout partition has the potential
to be unrepresentative and statistically misleading
Cross-validation and leave-one-out (LOO)
– Con: do not lead directly to a single production model
– Pro: use all available data for evaulation
– Pro: many partitions of data, helps average out
statistical variability