Lec 4 RQ
Lec 4 RQ
Rayleigh Quotients
• Applications
Rayleigh Quotients
Def 0.1. The Rayleigh quotient for a given symmetric matrix A ∈ S n (R)
is a multivariate function f : Rn − {0} 7−→ R defined by
xT Ax
f (x) = , x 6= 0.
xT x
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 3/32
Rayleigh Quotients
Remark. A Rayleigh quotient is always scaling invariant, that is, for any
nonzero vector x ∈ Rn ,
(kx)T A(kx) xT Ax
f (kx) = = = f (x), for all k 6= 0
(kx)T (kx) xT x
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 4/32
Rayleigh Quotients
n
Sn = {x ∈ R : kxk = 1},
kxk = 1
f |Sn (x) = xT Ax, x ∈ Sn .
Interpretation:
The Rayleigh quotient is essentially
a quadratic form over unit sphere.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 5/32
Rayleigh Quotients
!
1 3
Example 0.1. The Rayleigh quotient for A = ∈ S 2 (R) is
3 2
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 6/32
Rayleigh Quotients
We plot below the values of f along the circle xT x = x21 + x22 = 1 (left)
and also the full graph in 3 dimensions (right).
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 7/32
Rayleigh Quotients
max xT Ax
x∈Rn : kxk=1
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 8/32
Rayleigh Quotients
Theorem 0.1. For any given symmetric matrix A ∈ S n (R), let its largest
and smallest eigenvalues be λ1 and λn , with associated eigenvectors
v1 , vn ∈ Rn , respectively. Then the maximum (or minimum) value of the
T
associated Rayleigh quotient xxTAx x
is equal to the largest (or smallest)
eigenvalue of A, achieved by the corresponding eigenvectors:
xT Ax
max = λ1 , @ x = ±v1
x∈R : x6=0 xT x
n
xT Ax
min = λn , @ x = ±vn
x∈Rn : x6=0 xT x
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 9/32
Rayleigh Quotients
!
1 2
Example 0.2. For the PSD matrix A = , we have previously
2 4
obtained its eigenvalues and eigenvectors
1 1
λ1 = 5, λ2 = 0; v1 = √ (1, 2)T , v2 = √ (−2, 1)T
5 5
x21 +4x22 +4x1 x2
The associated Rayleigh quotient Q(x) = x21 +x22
has the following
extreme values:
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 10/32
Rayleigh Quotients
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 11/32
Rayleigh Quotients
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 12/32
Rayleigh Quotients
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 13/32
Rayleigh Quotients
For this goal, we need to know how to differentiate functions like xT Ax, kxk2
with respect to the vector-valued variable x.
We present a few formulas of such kind below (the proofs can be found in
the notes).
Proposition 0.2. For any fixed symmetric matrix A ∈ S n (R), matrix
B ∈ Rm×n and vector a ∈ Rn , we have
∂ T ∂
(a x) = a, (kxk2 ) = 2x
∂x ∂x
∂ T ∂
(x Ax) = 2Ax, (kBxk2 ) = 2BT Bx
∂x ∂x
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 14/32
Rayleigh Quotients
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 15/32
Rayleigh Quotients
xT Ax
max
x6=0 xT x +
vT
1 x=0
xT Ax
max
x6=0 xT x
vT T
1 x=v2 x=0
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 16/32
Rayleigh Quotients
xT Ax
max = λ2 (when x = ±v2 )
x6=0 xT x
vT
1 x=0
xT Ax
max = λ3 (when x = ±v3 )
x6=0 xT x
vT T
1 x=v2 x=0
and so on.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 17/32
Rayleigh Quotients
0 0 0
Example 0.3. Let A = 0 1 −1 ∈ S 3 (R). By direct calculation,
0 −1 1
this matrix has the following eigenvalues and eigenvectors
0 0 1
1 1
λ1 = 2, λ2 = λ3 = 0, v1 = √ 1 , v2 = √ 1 , v3 = 0
2 2
−1 1 0
T
Thus, the unrestricted Rayleigh quotient, f (x) = xxTAx
x
, has the maximum
value of λ1 = 2, which can be achieved at x = ±v1 .
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 18/32
Rayleigh Quotients
xT Ax
max = 0,
x6=0 xT x
vT
1 x=0
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 19/32
Rayleigh Quotients
xT Ax
f (x) = .
xT Bx
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 20/32
Rayleigh Quotients
This is also a function defined over R2 with the origin excluded, and scaling
invariant like ordinary Rayleigh quotients:
(kx)T A(kx) xT Ax
f (kx) = = = f (x), for all x 6= 0
(kx)T B(kx) xT Bx
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 21/32
Rayleigh Quotients
! !
2 3 2 3
Example 0.4. Given A = ∈ S 2 (R) and B = 2 (R),
∈ S+
3 2 3 5
we have the following generalized Rayleigh quotients:
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 22/32
Rayleigh Quotients
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 23/32
Rayleigh Quotients
Theorem 0.4. For any two matrices A ∈ S n (R) and B ∈ S+ n (R), let the
xT Ax
max = λ1 , @ x = ±v1
x6=0 xT Bx
xT Ax
min T = λn , @ x = ±vn
x6=0 x Bx
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 24/32
Rayleigh Quotients
xT Ax
max
x6=0 xT Bx
is equivalent to the following constrained optimization problem:
max xT Ax subject to xT Bx = 1
x∈Rn
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 25/32
Rayleigh Quotients
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 26/32
Rayleigh Quotients
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 27/32
Rayleigh Quotients
! !
2 3 2 3
Example 0.5. Consider the two matrices A = and B = ,
3 2 3 5
where A is symmetric and B is positive definite. We have already solved
the generalized eigenvalue problem (A, B) previously:
! !
1 1 1 3
λ1 = 1, λ2 = −5, and v1 = √ , v2 = √ .
2 0 2 −2
T
Thus, by the preceding theorem, the generalized Rayleigh quotient xxT AxBx
has a maximum value of λ1 = 1 and a minimum value of λ2 = −5, attained
at the corresponding generalized eigenvectors, ±v1 , ±v2 , respectively.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 28/32
Rayleigh Quotients
xT Ax
max = λ2 (when x = ±v2 )
x6=0 xT Bx
vT
1 Bx=0
xT Ax
min = λ3 (when x = ±v3 )
x6=0 xT Bx
vT T
1 Bx=v2 Bx=0
and so on.
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 29/32
Rayleigh Quotients
0 0 0
Example 0.6. Let A = 0 3 1 ∈ S 3 (R), B = diag(1, 2, 2) ∈ S+ 3 (R).
0 1 3
By direct calculation, the generalized eigenvalues and eigenvectors of
(A, B) are
0 0 1
1 1
λ1 = 2, λ2 = 1, λ3 = 0; v1 = 1 , v2 = 1 , v3 = 0
2 2
1 −1 0
T
Thus, the unrestricted generalized Rayleigh quotient, f (x) = xxT Ax
Bx
over
n
R − {0}, has the maximum value of λ1 = 2, which can be achieved at
x = ±v1 .
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 30/32
Rayleigh Quotients
If we now exclude v1 from the optimization domain (and consider only the
orthogonal complement of it), by the preceding theorem, the maximum
value of f changes to λ2 = 1:
xT Ax
max = 1,
x6=0 xT Bx
vT
1 Bx=0
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 31/32
Rayleigh Quotients
vT Sb v
• LDA: maxv6=0 vT Sw v
(Sb : between-class scatter, Sw : within-class
scatter)
vT Lv
• Laplacian Eigenmaps (and spectral clustering): min v6=0 vT Dv
vT D1=0
(L: graph Laplacian matrix, D: degree matrix)
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 32/32