DimensionalityReduction 13022024
DimensionalityReduction 13022024
DIMENSIONALITY
REDUCTION
Dimensionality of input
2
Supervised
Train using selected subset
Estimate error on validation data set
Unsupervised
Look at input only(e.g. age, income and savings)
Select subset of 2 that bear most of the information
about the person
Mutual Information
8
Forward search
Start from empty set of features
Try each of remaining features
Estimate classification/regression error for adding specific
feature
Select feature that gives maximum improvement in
validation error
Stop when no significant improvement
Backward search
Start with original set of size d
Drop features with smallest impact on error
Floating Search
10
Floating search
Two types of steps: Add k, remove l
More computations
Feature Extraction
11
Taking Derivatives
z = WT(x – m)
where the columns of W are the eigenvectors of ∑,
and m is sample mean
Centers the data at the origin and rotates the axes
How to choose k ?
19
1 2 k
1 2 k d
V is d x k matrix (k<d)
m1 m2
2
J w
s s
2
1
2
2
Summary
32
Feature selection
Supervised: drop features which don’t introduce large
errors (validation set)
Unsupervised: keep only uncorrelated features (drop
features that don’t add much information)
Feature extraction
Linearly combine feature into smaller set of features
Supervised
PCA: explain most of the total variability
FA: explain most of the common variability
Unsupervised
LDA: best separate class instances