Professional Documents
Culture Documents
Principal Component Analysis (PCA) - : San José State University Math 253: Mathematical Methods For Data Visualization
Principal Component Analysis (PCA) - : San José State University Math 253: Mathematical Methods For Data Visualization
Introduction
• Many data sets have very high dimensions nowadays, causing signifi-
cant challenge in storing and processing them.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 2/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 3/49
Principal Component Analysis (PCA)
A demonstration
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 4/49
Principal Component Analysis (PCA)
Another example
0.5
0.4
0.3
0.2
0.1
0
0 100 200 300 400 500 600 700 800
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 5/49
Principal Component Analysis (PCA)
x1 , . . . , xn ∈ Rd , find a line S b
b
b x = tv + b′
xi
parametrized by x(t) = t · v + b b
b
b
ai
onal projections of the data onto the × b
b
line b b b
b b
b
PS (xi ) = v vT (xi − b) +b b
| {z }
:=ai b
= ai v + b, 1≤i≤n
have the largest possible variance.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 6/49
Principal Component Analysis (PCA)
Mathematical formulation
First observe that for parallel lines, the projections are different, but the
amounts of variance are the same! ←− This implies that the choice of b
is not unique.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 7/49
Principal Component Analysis (PCA)
We have thus eliminated the variable b from the problem, so that we only
need to focus on the unit-vector variable v (representing the direction of
the line).
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 8/49
Principal Component Analysis (PCA)
= vT Cv.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 9/49
Principal Component Analysis (PCA)
max vT Cv
v:kvk=1
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 10/49
Principal Component Analysis (PCA)
This shows that v2 , v3 etc. are the next best orthogonal directions.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 11/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 12/49
Principal Component Analysis (PCA)
The total scatter of the k-dimensional PCA projections is equal to the sum
of the scatter onto each direction. We prove this for the case of k = 2:
X X X X
k(ai , bi ) − (0, 0)k2 = (a2i + b2i ) = a2i + b2i = λ1 + λ2
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 13/49
Principal Component Analysis (PCA)
• The projection of the data point xi onto vj , i.e., vjT (xi − x̄), is
called the jth principal component of xi .
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 14/49
Principal Component Analysis (PCA)
Example 0.1. Perform PCA (by hand) on the following data set (rows
are data points):
1 −1
−1 1
X= .
2 2
−2 −2
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 15/49
Principal Component Analysis (PCA)
Computing
C ≈ Vk Λk VkT
which can be a significant challenge for large data sets in high dimensions.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 16/49
Principal Component Analysis (PCA)
x1T e 1T
x
.
. n×d and Xe = .. ∈ Rn×d (where x
Let X = . ∈R . e i = xi − x̄) be
xnT x
en T
the original and centered data matrices (rows are data points).
Then
xe 1T
X
e iT = [x
en] ·
e1 . . . x
.. eT e
C= x
eix . = X X.
x
enT
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 17/49
Principal Component Analysis (PCA)
eTX
C=X e ∈ Rd×d .
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 18/49
Principal Component Analysis (PCA)
Interpretations:
e = U · Σ · VT = UΣ · VT
X
Then
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 19/49
Principal Component Analysis (PCA)
To see the last one, consider any principal direction vj . The corresponding
principal component is
e j = σ j uj
Xv
with scatter λj = σj2 .
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 20/49
Principal Component Analysis (PCA)
x = x̄ + Vk α
• Projections of the data onto this plane are given by the rows of
e k VT
PS (X) = 1x̄T + XV k
| {z }
Y
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 21/49
Principal Component Analysis (PCA)
Steps:
e = [x1 − x̄, . . . , xn − x̄]T where x̄ = 1 P
1. Center data: X n xi
e ≈ Uk Σk VT
2. Perform rank-k SVD: X k
3. Return: Y = XV
e k = Uk Σk
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 22/49
Principal Component Analysis (PCA)
planes coincide: b b
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 23/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 24/49
Principal Component Analysis (PCA)
Xtilde = X - mean(X,1);
[U,S,V] = svds(Xtilde, k); % k is the reduced dimension
Y = Xtilde*V;
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 25/49
Principal Component Analysis (PCA)
b
b b b
b
b b b b
b
b b
b b
b
b f
b b
pc1 (Xv 1)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 26/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 27/49
Principal Component Analysis (PCA)
4-6
4 5 5
3 4
4
3
2 3
2
1 2
1
0 1
0
-1 0
-1
-2 -1
-2
-3 -2
-3
-4 -3
-4
-5 -4
-5
-5 -4 -3 -2 -1 0 1 2 3 4 -4 -2 0 2 4 6 -6 -5 -4 -3 -2 -1 0 1 2 3 4
7-9
4
4
6
3
3
2
2 4
1
1
0
0 2
-1
-1
0 -2
-2
-3
-3
-2 -4
-4
-5
-5
-4
-6
-6 -4 -2 0 2 4 -5 -4 -3 -2 -1 0 1 2 3 4 5 -6 -5 -4 -3 -2 -1 0 1 2 3 4
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 28/49
Principal Component Analysis (PCA)
Common values of p are .95 (the most commonly used), or .99 (more
conservative, less reduction), or .90, .80 (more aggressive).
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 29/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 30/49
Principal Component Analysis (PCA)
• very popular!
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 31/49
Principal Component Analysis (PCA)
This method is called factor analysis and its solution can be derived by
using the MLE approach.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 32/49
Principal Component Analysis (PCA)
y = VT (x − x̄)
For nonlinear data, PCA will need to use a dimension higher than the
manifold dimension (in order to preserve most of the variance).
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 33/49
Principal Component Analysis (PCA)
PCA
b b
b b b b b b
b b b b b b b b
b b b b b b b b
b b b b b b
b b b b b b
b b
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 34/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 35/49
Principal Component Analysis (PCA)
Theorem 0.2. Let X ∈ Rn×d be the given data set and k ≥ 1 an integer.
The best k-dimensional plane through the origin for fitting the data is
spanned by the top k right singular vectors of X:
X
|{z} ≈ X k = Uk Σk Vk
|{z} | {z } |{z}
given data projections coefficients basis
min kX − XVVT k2
V∈Rd×k : VT V=I k
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 36/49
Principal Component Analysis (PCA)
1 1
x(t) = (2, 2) + t · √ , −√ , 1+ b
2 2
|
while the SVD line (best-fit line +1
| +2
| +3
|
through the origin) is
1 1
x(t) = t · √ ,√ .
2 2
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 37/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 38/49
Principal Component Analysis (PCA)
Summary information:
A significant challenge:
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 39/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 40/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 41/49
Principal Component Analysis (PCA)
2. Remove words that occur either in exactly one document (rare words
or typos) or in “too many“ documents (stopwords or common words)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 42/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 43/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 44/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 45/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 46/49
Principal Component Analysis (PCA)
We also display the top 20 words that are the most “relevant” to the
underlying topic of each class.
The following score can then be used to measure and compare the relevance
of each keyword:
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 47/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 48/49
Principal Component Analysis (PCA)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 49/49