Professional Documents
Culture Documents
SMSP
SMSP
Smoothing Splines
Tongtong Wu
Feb 29, 2004
Outline
Introduction
Linear and polynomial regression, and
interpolation
Roughness penalties
Interpolating and Smoothing splines
Cubic splines
Interpolating splines
Smoothing splines
Natural cubic splines
Choosing the smoothing parameter
Available software
Key Words
roughness penalty
penalized sum of squares
natural cubic splines
Motivation
10
8
6
(y18)
4
2
5 10 15
Index
Motivation
10
8
6
y18
4
2
5 10 15
Index
Motivation
10
8
6
y18
4
2
5 10 15
Index
Motivation
10
8
6
(y18)
4
2
Spline(y18)
5 10 15
Index
Introduction
Linear and polynomial regression :
Global influence
Increasing of polynomial degrees happens in
discrete steps and can not be controlled
continuously
Interpolation
Unsatisfactory as explanations of the given
data
Roughness penalty approach
A method for relaxing the model
assumptions in classical linear regression
along lines a little different from
polynomial regression.
Roughness penalty approach
Aims of curving fitting
A good fit to the data
To obtain a curve estimate that does not
display too much rapid fluctuation
Basic idea: making a necessary
compromise between the two rather
different aims in curve estimation
Roughness penalty approach
Quantifying the roughness of a curve
An intuitive way:
g ' ' (t ) dt
b 2
a
(g: a twice-differentiable curve)
Motivation from a formalization of a
mechanical device: if a thin piece of
flexible wood, called a spline, is bent to
the shape of the graph g, then the leading
term in the strain energy is proportional to
2
g ' '
Roughness penalty approach
Penalized sum of squares
n
S ( g ) Yi g (ti ) g ' ' (t ) dt
2 b 2
a
i 1
g: any twice-differentiable function on [a,b]
: smoothing parameter (‘rate of exchange’
between residual error and local variation)
4
2
5 10 15
Index
Roughness penalty approach
Curve for a small value of
10
8
6
y18
4
2
5 10 15
Index
Interpolating and Smoothing Splines
Cubic splines
Interpolating splines
Smoothing splines
Choosing the smoothing parameter
Cubic Splines
Given a<t1<t2<…<tn<b, a function g is a
cubic spline if
1. On each interval (a,t1), (t1,t2), …, (tn,b), g is a
cubic polynomial
2. The polynomial pieces fit together at points ti
(called knots) s.t. g itself and its first and
second derivatives are continuous at each ti,
and hence on the whole [a,b]
Cubic Splines
How to specify a cubic spline
g (t ) d i (t ti )3 ci (t ti ) 2 bi (t ti ) ai for ti t ti 1
Natural cubic spline (NCS) if its second
and third derivatives are zero at a and b,
which implies d0=c0=dn=cn=0, so that g is
linear on the two extreme intervals [a,t1]
and [tn,b].
Natural Cubic Splines
Value-second derivative representation
We can specify a NCS by giving its value
and second derivative at each knot ti.
Define g ( g1 ,, g n )' , where gi g (ti )
( 2 ,, n1 )' , where i g ' ' (ti )
which specify the curve g completely.
However, not all possible vectors
represent a natural spline!
Natural Cubic Splines
Value-second derivative representation
Theorem 2.1
The vector g and specify a natural
spline g if and only if
Q ' g R
Then the roughness penalty will satisfy
b
a
g ' ' (t ) 2 dt ' R g ' Kg
Natural Cubic Splines
Value-second derivative representation
h11 0 0 hi ti 1 ti for i 1,, n
1 1
h1 h2 h21 0
h21 h21 h31 0
Q
0 h31 0
1
0 0 hn 1 n( n 2 )
1 1
( h
3 1 3 h ) h2 0
6
1 1
R h 2 (h2 h3 ) 0
6 3
0 0
1
(hn 2 hn 1 )
3 ( n 2 )( n 2 )
Natural Cubic Splines
Value-second derivative representation
R is strictly diagonal dominant, i.e.
| rii | j i | rij |, i
R is positive definite, so we can define
K QR 1Q'
Interpolating Splines
To find a smooth curve that interpolate (ti,zi),
i.e. g(ti)=zi for all i.
Theorem 2.2
Suppose n 2 and t1<…<tn. Given any
values z1,…,zn, there is a unique natural cubic
spline g with knots ti satisfying
g (ti ) zi for i 1,, n
Interpolating Splines
The natural cubic spline interpolant is the
unique minimizer of g ' '2 over S2[a,b] that
interpolate the data.
Theorem 2.3
Suppose g is the interpolant natural cubic
~ S [a, b] with g~(t ) z for i 1,, n
spline, g 2 i i
then
~ ' '2 g ' '2
g
Smoothing Splines
Penalized sum of squares
n
S ( g ) Yi g (ti ) g ' ' (t ) dt
2 b 2
a
i 1
g: any twice-differentiable function on [a,b]
: smoothing parameter (‘rate of exchange’
between residual error and local variation)
S ( g ) S ( g~)
Smoothing Splines
2. Existence and uniqueness
Let Y (Y1 ,, Yn )' then
n
i
Y g (t i )2
(Y g )' (Y g )
i 1
since g be precisely the vector of g (ti ) .
Express g ' ' 2
g ' Kg ,
S(g) (Y g)'(Y g) g'Kg
g'(I K)g 2Y' g Y'Y
Minimum is achieved by setting g ( I K ) 1Y
Smoothing Splines
2. Theorem 2.4
Let ĝ be the natural cubic spline with
knots at ti for which g ( I K ) Y . Then
1
Generalized cross-validation
n
i i
Y ˆ
g (t ) 2