Professional Documents
Culture Documents
Esbensen K Swarbrick B Westad F Anderson M Whitcomb P Multiv
Esbensen K Swarbrick B Westad F Anderson M Whitcomb P Multiv
6th Edition
An introduction to Multivariate Analysis, Process
Analytical Technology and Quality by Design
Kim H. Esbensen
and
Brad Swarbrick
Norway
CAMO Software AS
Gaustadalléen 21
N-0349 Oslo, Norway
Tel: (+47) 223 963 00
United States
CAMO Software Inc.
33300 Egypt Lane
Suite L100
Magnolia, TX 77354, United States
Phone: +1 732 726 9200
ISBN 978-82-691104-0-1
© 2018 CAMO Software AS
All Rights reserved. No part of this publication may be reproduced, stored or transmitted, in
any form or by any means, except with the prior permission in writing of the publishers.
6th Edition
Preface
Chapter 5: Preprocessing
5.1 Introduction
5.2 Preprocessing of discrete data
5.2.1 Variable weighting and scaling
5.2.2 Logarithm transformation
5.2.3 Averaging
5.3 Preprocessing of spectroscopic data
5.3.1 Spectroscopic transformations
5.3.2 Smoothing
5.3.3 Normalisation
5.3.4 Baseline correction
5.3.5 Derivatives
5.3.6 Correcting multiplicative effects in spectra
5.3.7 Other general preprocessing methods
5.4 Practical aspects of preprocessing
5.4.1 Scatter effects plot
5.4.2 Detailed example: preprocessing gluten–starch mixtures
5.5 Chapter summary
5.6 References
Figure 1.1: Matrices for the two types of measurements X and Y, made on the same set of n
objects.
Variance
Standard Deviation
The reader will find many examples in the scientific and technical
literature of the use of the “strength” of correlation, often expressed
by the correlation coefficient r, and it is not infrequent to see claims,
for example, that r being of the order of 0.50 is testimony to a “good
correlation”. However, this says that only 25% of the total scaled
covariance is systematic, see the right-most panel in Figure 1.2; a
correlation coefficient squared of 0.261 is anything but “good”. One
should always use r2 as a measure of the fraction of the total
correlation as an expression of a systematic relationship—never r.
2.1 Terminology
In the statistical literature, the concepts of population and sample are
central. A population is taken to mean a large (usually theoretical)
collection of observation values (in theory all possible values
pertaining to the context). A sample, in the statistical context, is a
small selected set of observations, taken from a population and
which is assumed to be able to represent that population. For
historical reasons the terminology used in statistics and in the Theory
of Sampling (TOS), refer to chapter 3, overlap for a few critically
important concepts, that can cause serious confusion if not defined
very clearly, foremost of which are the terms “sample” and
“sampling”. It is necessary to distinguish clearly between what can be
termed a samplestat vs a sampleTOS, (the first is a set of values, the
second is a physical sample extracted from a predecessor lot)—and
samplingstat vs samplingTOS (the sampling process employed in the
statistical and the physical sampling contexts respectively); these
issues are laid out more fully in chapter 3. In the present chapter,
usage of sample is mostly in the statistical sense unless carefully
noted.
The branch of statistics known as parametric statistics is founded
on the properties of the so called Normal Distribution. The normal
distribution (to be discussed in more detail in section 2.4) is described
by two parameters, the mean and the variance, hence where the
name parametric statistics comes from. The normal distribution is
described by a bell-shaped curve where the majority of values of
observations are concentrated around the centre of the curve and
when moving away from the centre, in both directions, the
concentration of observations decreases rapidly. When the values of
the mean and the variance are calculated from a small sample of
observations, these values are called statistics rather than
parameters.
The normal distribution forms much of the basis of Inferential
Statistics, i.e. the ability to make inferences (or deductions) about the
large data set, the population, based on a small, but well understood
sample taken from the entire population.
Table 2.1: Terminology and symbols used in parametric statistics.
Parameter μ σ2 σ
Statistic s2 s
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0], i.e. a data set where N = 6
where σ2, s2, μ and are described in Table 2.1, i represents the
particular x value being summed and N is again the total number of
observations. Note in equation 2.2b, the denominator is N – 1 rather
than N. The term N – 1 is known as the degrees of freedom (dof) of
the data and this will be discussed further in section 2.5.4.
The histogram
A Histogram is a special form of bar chart that plots the frequency of
occurrence of values falling in pre-defined data intervals, i.e. the data
range is divided up into a specified number of “bins” of particular
width, and the height of each bin bar is a reflection of how many
times data values lie in these bins. The histogram serves the purpose
of providing a graphical overview of the spread of a data set. A
general histogram is provided in Figure 2.3.
The following questions can be answered using a histogram:
Figure 2.3: The histogram of a normally distributed data set.
Figure 2.4: The most often encountered types of univariate data distributions.
1. Are the data spread evenly over a wide range of the variable
outcome space, or are they concentrated into only a small region?
2. Do the data form one coherent group, or do they form two, or
possibly more groups?
3. Are the data symmetrically distributed around a central point (i.e.
normally distributed) or are they skewed to the left or to the right?
(See further below.)
The human cognitive system is highly adept at detecting patterns
when data are displayed in simple graphical formats such as the
histogram. Figure 2.4 shows some of the most commonly
encountered types of univariate distributions.
To compliment Figure 2.4, a few examples of such data structures
and their typical meanings are provided below.
Normal: A quality attribute of a tightly controlled process should
show a normal distribution of data with a mean value close to the
expected target value and a small spread, indicating tight control. In
regression analysis, the spread of the so called residual values around
zero, with a small variance is a requirement of least squares fitting.
Left Skewed: This type of distribution can be expected, for example,
when a system is under development and has not yet been
optimised. The target value lies to the right in the histogram and may,
for example, represent overall yield. The “tail” of the data tapers off to
the left of the graph, hence the name. The values to the left may
represent instances where the desired yield was not achieved.
Typical: This is the common shape of most experimental data and
may sometimes (but not necessarily always) represent a situation
somewhat close to the normal distribution.
Right Skewed: This is the usual situation, for example, when testing
for impurities in finished products. The majority of the measurements
are to the left of the plot, with the tail skewed to the right. The tail
indicates occurrences where high impurities are the exception, rather
than the norm.
Bimodal: This is the typical situation when two populations may exist
in the same data set.
Boxcar: This represents an even distribution of measurements over
the entire range. It is the desired distribution of data for multivariate
calibration development purposes discussed in chapter 7. A boxcar
distribution has a small value of kurtosis (“peakedness”), usually less
than 3; this is described in more detail below in this section.
For a small number of samples N, the so-called “small sample”
case in statistics, the shape of a histogram can be very misleading. It
has been suggested that histograms are best used when the number
of samples is greater than 75, Montgomery [4]. The reader will
undoubtedly quickly get a feeling for how, and to what extent, a
small-sample basis serves proper statistics, or not…
Box plots
A Box Plot (sometimes known as a “box and whisker” plot)
represents the distribution of a particular data set in terms of its
percentiles, separating the data into quartiles as illustrated in Figure
2.9.
Figure 2.9: The Box Plot and its graphic quartile distribution subdivisions.
Fifty percent of the data are contained within the central box part
of the plot while the extreme values (represented by the Whiskers)
contain the outer fifty percent. The horizontal line found in the box
part of the plot represents the median value.
The case where the whiskers are of equal size, and the median
lies in the centre of the box, is a clear sign that the data set is
symmetrically distributed (i.e. possibly normally distributed—but only
the kurtosis will tell). The vertical length (size) of the inner box and of
the whiskers represents the overall spread of the data. When the
sizes of the whiskers are different from each other, the longest
whisker signifies skewness in the data.
The box plot is a powerful tool for providing an initial overview of
data before further analysis. It finds major use when comparing the
uniformity and magnitudes of multiple variables in a single overview
plot. Figure 2.10 provides an example of a multiple box plot for
sensory preference data collected on raspberry samples used for jam
making.
Figure 2.10: Multiple box plots showing the relative distribution and median magnitudes of a
number of sensory variables measured and monitored in the jam making process.
After taking the above short detour into proper statistical thinking, the
main question to be asked is, “what is an acceptable risk level?”.
There is unfortunately no hard and fast rule for setting a value for α.
However, the most commonly used value is α = 0.05—this is the
canonical choice if there are no special arguments for other choices.
This value signifies 95% confidence of making a right decision in
terms of the type I error (compared to the risk of 5% of making a
wrong decision). The value 95% is obtained using the 100(1 – α)%
criterion. Some other commonly used significance levels are 0.25, 0.1
and 0.01 representing 75%, 90% and 99% confidence levels,
respectively.
A p-value obtained for a particular test must be compared with
the significance level chosen. Table 2.3 provides some general rules
for interpreting the p-value at the canonical α level 0.05.
The concept of p- and α-values can be illustrated graphically as
shown in Figure 2.11; these snapshots of fundamental statistical
concepts have been found useful for scores of entry-level data
analysts for decades, as they embody the essential issues in a
remarkably simple format.
Acknowledging that around the mean, the probability of a
samplestat value in this vicinity is close to 1, then, as one moves
further and further away from the mean, the p-value approaches zero.
This means that there must be some point at which a decision must
be made as to whether the observation value (measurement) belongs,
or not, to the population defined by the null hypothesis. At α = 0.05,
there is an acceptance that there is a 5% (a 1 in 20) chance of being
wrong. If the p-value calculated is >0.05 this is indicative of a sample
belonging to the population defined by the null hypothesis. This is
another compelling argument to be in full command regarding
variance, i.e. always try to get measurement variances (indeed also
the sampling variabilities) estimated as validly as possible. Indeed,
the mandate is always work diligently to curtail all influential variance-
inducting phenomena and processes: it is often possible to reduce
both sampling and measurement errors (variances) significantly,
provided one is in the disposition of the necessary competences, see
chapters 3, 8 and 9. Work of this kind cannot fail to give desirable
results a.o. in the form of increased statistical test power.
>0.1 Insignificant
0.01–0.05 Significant
Figure 2.12: The difference in risk for one-sided and two-sided hypothesis tests.
Confidence intervals
As the name suggests, a confidence interval is a statement of how
confident an analyst is that a particular value lies within some
tolerance (interval). There is a confidence, to some level of probability,
α, that the value x lies within a lower bound, defined as L and an
upper bound, defined as U. The 100(1 – α)% confidence interval is
defined by the alternative test types (one-sided or two-sided) and the
probability level (risk) that is willing to be accepted. The smaller the α
value, the wider the confidence interval, there is a willingness to
accept more type II errors. General statements of the confidence
interval are provided as follows and these are discussed in more
details in section 2.5.5 when introducing the t-test.
Two-Sided Confidence Interval: L ≤ μ ≤ U
One-Sided Confidence Interval: L ≤ μ or μ ≥ U
Here is an example to illustrate how to set up an appropriate
statistical test scenario: it is the purpose of the test to see if the mean
value of a samplestat is equivalent to some specified target value. Now
the novice “statistician” gets to work, and formulates this wish so as
to match the appropriate test framework: Equivalence is, in this case,
defined such that if the target value lies within the confidence interval
of the sample mean, there is no statistical reason to consider the
sample as being different from the target. Furthermore, if α = 0.05 is
chosen and the target value lies within the confidence interval of the
sample, there is a 95% level of confidence that the sample meets the
target value specification. This type of (re-)formulation of the practical
test objective into the framework of a standard statistical test and all
the, at first sight, intricate paraphernalia of risk, confidence, error
types and what-not, will take a lot of hard work if met with for the very
first time—and there is no easy fix. There are insights, experiences
and competences that can only be obtained by hard work. Do not
give up; there is a world of real power in mastering statistical
hypothesis testing (“it was all your competitors that gave up too
early…”).
where μ represents the mean of the empirical data set and σ is the
best possible estimate of the sample standard deviation. The
transformed variables are then placed in order, ranked, from lowest to
highest values and the expected frequency value (expressed as 1/n,
where n = number of samples) are plotted for each transformed x-
variable as a stepped cumulative distribution plot. The maximum
vertical distance between each point in the step curve and the
theoretical distribution is calculated and compared to a K–S table. If
this distance exceeds the tabulated distance at a specified probability
level (usually p = 0.05), the null hypothesis, that the data come from a
normal distribution, must be rejected in favour of the alternative
hypothesis. An example output is provided in Figure 2.13.
A correction commonly applied when using the K–S method is the
“Lilliefor’s correction”. It is used to test the null hypothesis that the
data come from a normal population, when the null hypothesis does
not specify which normal distribution, i.e. does not specify the
expected value (the mean) and the variance. This test procedure
follows the original K–S procedure closely, and only differs in the way
it calculates the test statistic for assessing normality. For more details
on this procedure, refer to Miller and Miller [15].
In the case where a data set is found to be non-normally
distributed, there exists a family of power transformations that can be
used to transform non-normally distributed data into normally
distributed data for the purposes of statistical comparison. Two
common examples are the Box–Cox transformation, Draper and
Smith [16], and the Johnson Transformation, Chou et. al. [17]. Care
should be taken when using such transformations as the underlying
parent distribution is non-normal. This is outside the scope of this
introductory level and the interested reader is referred to the literature
for more information.
Figure 2.13: An example of the cumulative distribution output associated with the K–S test.
Table 2.4: Quality measures (masses) from process data for Example
2.1.
Sample number Mass (g)
1 94.1
2 93.2
3 91.4
4 95.1
5 92.4
6 91.2
7 95.2
8 94.3
9 93.2
10 95.3
For example, say samplesstat are being drawn from a right skewed
distribution and it is decided to take five measurements at a time. By
taking the average of these values, even if there is one extreme value
in the set, their average will usually be closer to the majority of the
sample values than the extreme. By iterating this process numerous
times, this will result in new subset average values that are nearly
normally distributed. The number of times the parent distribution has
to be sampled to obtain normality is defined by the Central Limit
Theorem (CLT). The interested reader is referred to chapter 8* and the
literature for more details of the CLT, Montgomery [4].
Nevertheless, when a parent distribution is sampled many times,
the averaged values from these sub-samplestat averages provide an
analyst with a much more confident estimate of the population mean,
with respect to “grabbing” just one value, at random. If the sample
size is n ≥ 5, there will definitely be more confidence in the values of
the variance observed, i.e. as manifested by the standard deviation of
the spread of the individual samplestat means. The value of n ≥ 5 has
been stated as a sufficient number of samples in order to provide a
reliable estimate of variance Massart et al. [18], but it is fair to say that
there is not universal agreement about this very small N.
Observe that this dispute, along with several others (all related to
“what is the minimum N needed in order to…”), often reflects the
evergreen desire to get away with the current task in data analysis
and statistics with the least effort—but even the present cursory
introductions should have made it abundantly clear that there is no
magical “small number, N” that will fix such problems.
The standard error (s.e.) (often called the standard error of the
mean) is defined as the confidence level regarding the reliability of
that mean value actually representing the population mean. It is
defined by equation 2.13.
Figure 2.17: Cumulative distribution of the re-samples subset average values showing the
characteristics of a normal distribution.
Figure 2.18: Schematic differences in variance between two data sets exhibiting identical
means.
Table 2.8: Mass results for two samples taken from a manufacturing
process.
Sample number Mass (g) (pre improvement) Mass (g) (post improvement)
1 94.1 93.4
2 93.2 92.9
3 91.4 93.5
4 95.1 94.8
5 92.4 93.7
6 91.2 92.4
7 95.2 94.1
8 94.3 93.7
9 93.2 92.8
10 95.3 93.5
F0 4.949
F0.05,9,9 3.179
p-value 0.026
t-test Used for “small sample” cases (usually between 5 and 30)
Uses the t-table which scales the distribution based on the
number of measurements
Requires no prior knowledge of the population variance
Figure 2.20: The t-distribution is scaled based on the number of samples in the set. When
this number approaches 30, the t-distribution is equivalent to the Z-distribution.
1. Test for normality of the observations (by applying the K–S test).
2. Test for equivalence of variance (F-test).
3. Apply the appropriate t-test (described in the following sections).
Figure 2.21: For the mean value of a samplestat to be effectively equivalent to the target
value, the target should lie in the confidence interval of the estimated sample mean.
Note that this is a two-sided test because there is interest in
testing the following hypotheses,
H0: = μ0
Ha: ≠ μ0
The value α/2 indicates that the test is two-sided and therefore at
95% confidence, the 2.5% tails of the t-distribution are being tested
(refer to Figure 2.12). Since there is only one sample, ν represents the
degrees of freedom for the sample set and is equal to N – 1. For this
example, the number of samples is 10, therefore the dof is 9 and the
critical t-value has the form t0.025,9. Table 2.12 provides the details of
the calculation.
The null hypothesis can only be rejected if |t0| > tα/2,ν. In this case,
since 3.04 > 2.262, it was concluded that the mean value of the
sample set does not meet the specification for this product. The
calculated p-value also suggests that the null hypothesis should be
rejected, i.e. p < 0.05.
N 10
t0 –3.04
tα/2,ν 2.262
p-value 0.015
92.3 ≤ μ0 ≤ 94.8
Table 2.13: Relevant part of the table of critical t-values for Example
2.3.
Since the specified value of 95 g does not lie within the 95%
confidence interval of the sample mean, the null hypothesis that the
sample mean is equal to the specified mean value is rejected in
favour of the alternative hypothesis: these means are statistically
different.
or
Ha: 1 < ,
2 1 > 2 (one-sided)
A p-value > 0.05 (or |t0| < tcrit) indicates that the null hypothesis
cannot be rejected, i.e. there is no difference between 1 and 2. A p-
value < 0.05 (or |t0| > tcrit) suggests that the sets of observations are
significantly different and therefore the null hypothesis must be
rejected.
Sample number Mass (g) (post improvement 1) Mass (g) (post improvement 2)
1 93.4 94.2
2 92.9 95.1
3 93.5 93.3
4 94.8 94.2
5 93.7 93.8
6 92.4 95.1
7 94.1 94.3
8 93.7 92.9
9 92.8 93.7
10 93.5 94.7
Difference 1 – 2 0.65
n1 10
n2 10
t0 –2.063
tα/2,ν 2.101
p-value 0.054
Example 2.4: Again, using the data from Example 2.3, a second set
of mass data was collected on the improved process to determine
whether the changes made a real impact or whether they were just
the result of chance. The data are provided in Table 2.14 and the
calculations for the t-test are provided in Table 2.15.
The first step is to establish normality and equivalence of variance
of the data sets. This was performed using the K–S and F-tests and
concluded that the equal variance assumption t-test can be used.
It is important to note that in this case the p-value is marginal (i.e.
between 0.05 and 0.1, refer to Table 2.3) so a decision must be made
as to decide whether a significant difference exists or not. This is a
case where the difference between statistically significant and
practically significant must be made, i.e. is the mean value of 93.5 g
really different from 94.1 g? So much for the power of properly
conducted statistical hypothesis tests—but perhaps the more
interesting real-world question here is: “Why are the data not centred
around 95 g, the specified process target?”.
Figure 2.22: Situations describing equivalent and non-equivalence of sample means using
the two-sample t-test.
t0 0.882
t0.025,9 2.262
p-value 0.401
Figure 2.24: An example time series plot with a histogram. The horizontal lines are control
limits.
Figure 2.25: Example of a time series plot showing a step change in the process. Note that
the marginal histogram (and its fitted distribution) is now non-normal.
If, for instance, the data showed trending, a step change or even
cyclic behaviour, static measures are not useful in detecting when
such changes occurred. The marginal histogram can only summarise
the final data structure. The data in Figure 2.25 shows the effect of a
step change in a process. Note that the distribution is now skewed,
indicating a change in the process, but it cannot locate when this
event happened. This can only be addressed using the time series
plot.
There is a direct link between hypothesis testing and time series
plots when applied to process or other data collected over time. The
time series chart is sometimes known as a run chart, where it is
expected that data is collected at regular time intervals. In a run
chart, the mean is usually plotted as a horizontal line and the variation
of the data around the mean can be visualised and further
investigated.
Time Series Analysis (TSA) [22, 23] is used to forecast the
possibility of a future event. It uses techniques such as linear
regression (chapter 7) and methods such as smoothing and de-
trending (chapter 5) to isolate the main patterns in the data. From
these patterns, models are developed that best describes future
events. This, of course, is highly dependent on the reliability and the
representativity of the historical data used to develop the model.
Therefore, TSA models are usually updated on a regular basis. Figure
2.26 provides a time series plot of what is known as cyclic data
(periodic data). Cycles may be due to many behind-the-scene
influences, e.g. they may reflect seasonal changes—or may be based
on external drivers that occur due to inflation and deflation of
economies over certain time periods. Again, if a histogram is fitted to
the cyclic data, it only provides a static indication of the variability of
the data as a whole and cannot detect the cycles in the data.
Figure 2.27 provides the same data as Figure 2.26, however, this
time the smoothed time series is plotted together with the original
data. Smoothing very often helps to better visualise the cyclic nature
of the data.
Regarding time series plots in general: if the data are
symmetrically (or near symmetrically) distributed around their mean
value in a random fashion, for example similar to the data in Figure
2.24, a certain level of confidence that the data are near normally
distributed around the mean would be perceived. The variation, or the
spread of the data with respect to the mean would provide an
indication of how close the overall data are to this mean. In process
applications, this is of primary importance as it is the goal of any
company to produce goods with as little variability as possible. If the
mean value of the data set is close to a specified target, it could be
said that the process, or system, is behaving as expected (the system
is “capable”). This is very similar to the scope of the problem
discussed in section 2.5.5 on one-sample t-tests.
Figure 2.27: Plot of original time series data (blue) with the smoothed data (red) overlaid to
better visualise the cyclic nature of the data.
Figure 2.29: An example of a control chart showing warning and action limits.
Figure 2.30: Individual control charts of temperature and pH showing no indication of out of
control situation.
Figure 2.31: The covariance structure of temperature and pH. This highlights the process
failure in a multivariate data space.
2.9 References
[1] George, L.M., Rowlands, D. Price, M. and Maxey, J. (2005).
Lean Six Sigma Pocket Toolbook, McGraw Hill.
[2] Brue, G. and Luansby, R.G. (2003). Design for Six Sigma,
McGraw Hill.
[3] Yang, K. and El-Haik, B. (2003). Design for Six Sigma, A
Roadmap for Product Development, 1st Edn, McGraw Hill
Professional.
[4] Montgomery, D.C., (2005). Introduction to Statistical Quality
Control, 5th Edn, John Wiley & Sons.
[5] Conover, W.J. (1998). Practical Non-Parametric Statistics, 3rd
Edn, John Wiley & Sons.
[6] Hollander, M. and Wolfe, D.A. (1999). Non-Parametric Statistical
Methods, 2nd Edn, Wiley Interscience.
[7] Box, G.E.P., Hunter, W.G. and Hunter, J.S. (1978). Statistics for
Experimenters, John Wiley & Sons.
[8] Muzzio, F.J., Robinson, P., Wightman, C. and Brone, D. (1997).
“Sampling practices in powder blending”, Int. J. Pharm. 155,
153–178. https://doi.org/10.1016/S0378-5173(97)04865-5
[9] Esbensen, K.H., Roman-Ospino, A.D., Sanchez, A. and
Romanach, R.J. (2016). “Adequacy and verifiability of
pharmaceutical mixtures and dose units by variographic
analysis (Theory of Sampling) – A call for a regulatory paradigm
shift”, Int. J. Pharm. 499, 156–174.
https://doi.org/10.1016/j.ijpharm.2015.12.038
[10] Esbensen, K.H., Paasch-Mortensen, P. (2010). “Process
sampling: Theory of Sampling – the missing link in process
analytical technologies (PAT)”, in Process Analytical
Technology, Ed by K.A. Bakeev, John Wiley & Sons, pp. 37–80.
https://doi.org/10.1002/9780470689592.ch3
[11] Montgomery, D.C. (2001). Design and Analysis of Experiments,
5th Edn, John Wiley & Sons.
[12] Evans, J.R. (2007). Statistics, Data Analysis and Decision
Modeling, 3rd Edn, Pearson Prentice Hall.
[13] Anderson, T.W. and Darling, D.A. (1954). “A test of goodness-
of-fit”, J. Amer. Stat. Assoc. 49, 765–769.
https://doi.org/10.1080/01621459.1954.10501232
[14] Ryan T.A., Joiner B.L. (1976). Normal Probability Plots and Tests
for Normality, Technical Report, Statistics Department, The
Pennsylvania State University. Available from:
http://www.minitab.com/uploadedFiles/Shared_Resources/Documents/Arti
[15] Miller, J.N. and Miller, J.C. (2005). Statistics and Chemometrics
for Analytical Chemistry, 5th Edn, Prentice Hall.
[16] Draper, N.R. and Smith, H. (1998). Applied Regression Analysis,
3rd Edn, John Wiley & Sons.
https://doi.org/10.1002/9781118625590
[17] Chou, Y.M., Polansky, A.M. and Mason, R.L. (1998).
“Transforming non-normal data to normality in statistical
process control”, J. Qual. Technol. 30(2), 133–141.
[18] Massart, D.L., Vandeginste, B.G.M., Buydens, L.M.C., De Jong,
S., Lewi, P.J. and Smeyers-Verbeke, J. (1997). Handbook of
Chemometrics and Qualimetrics, Part A, Elsevier Science.
[19] Fisher, R.A. (1954). Statistical Methods for Research Workers,
Oliver and Boyd.
[20] Beyer, W.H. (1987). CRC Standard Mathematical Tables, 28th
Edn, CRC Press.
[21] Student (1908). “The probable error of a mean”, Biometrika 6(1),
1–25. https://doi.org/10.1093/biomet/6.1.1
[22] Montgomery, D.C., Jennings, C.L. and Kulahci, M. (2008).
Introduction to Time Series Analysis and Forecasting, John
Wiley & Sons.
[23] Box, G.E.P., Jenkins, G.M. and Reinsel, G.C. (2008). Time
Series Analysis, Forecasting and Control, 4th Edn, John Wiley &
Sons. https://doi.org/10.1002/9781118619193
[24] Shewhart, W.A. (1931). Economic Control of Quality of
Manufactured Product.
* Observe already now, however, that the CLT stipulates that this desirable and useful effect
is strongly dependent upon taking many samplesstat and that they each should be “large”,
i.e. N should ideally be (much) larger than five (sic.).
Chapter 3: Theory of Sampling (TOS)
3.2 Heterogeneity
Heterogeneity of stationary lots and materials has two fundamental
aspects: constitutional heterogeneity (CH) and distributional
heterogeneity (DH), which are sufficient to describe (and quantify) all
operative aspects of sampling. Only a few salient definitions are
needed.
The Constitutional Heterogeneity represents the heterogeneity
dependent on the physical or chemical differences between individual
lot units, which TOS terms “fragments”, but “grains” provides a more
useful image for the physical lot constituents, e.g. mineral grains,
seed grains, kernels, biological cells. Any given target to be sampled,
characterised by lot geometry, material type and state, and grain-size
distribution, exhibits a CH which is an inherent property of the lot
material. Thus, CH plays out its role as a summation of the between-
grain scale relationships in all lots. CH is a material-specific property;
CH can only be reduced by altering the physical state of the material
(by comminution, i.e. crushing the material to a smaller overall grain
size). The grain (the fragment*) is the smallest lot unit in the sampling
scenario.
The Distributional Heterogeneity complements this
characterisation by describing all aspects of heterogeneity dependent
upon the spatial distribution of all practical constituents in the lot, as
seen from the point of view of the operative sampling tool size
(volume/mass) used; this reference volume is called an increment.
This sampling unit could, for example, be a sampling scoop, but will
also take on many different forms in relation to lot type and size, for
example, a spatula, shovel, drill core, syringe, knife, vial, a front-
loader bucket…
The physical manifestations of DH are stratification, segregation
and/or local groups-of-fragment concentrations, i.e. “clumps” of
material with a significantly higher (or lower) analyte concentration
than the average lot concentration, aL (positive or negative “hot
spots”). DH can actively be reduced by using a suite of “correct”
sampling methods to be delineated further in this chapter. DH can
never be larger than CH (in a sense DH always constitutes a fraction
of CH) and CH can never be strictly zero. Depending on the purpose
and scale of sampling (the effective “scoop size”), DH may be
negligible, but it is never zero. TOS defines homogeneity as the
(theoretical) limiting case of zero heterogeneity. If a homogeneous
material actually did exist, sampling would not be needed—as all
sampling errors would be zero, because all “samples” would be
identical. Upon reflection, it should be obvious that homogenous
materials do not exist in nature, technology or industry, but the
concept of homogeneity is still very useful as an ideal attribute or
state in some theoretical deliberations. In practice, however, TOS
considers that all materials are indeed always heterogeneous, and
can consequently be treated in a unified manner. If one can deal
effectively with significantly heterogeneous matter, one can deal with
any type of material—opening up for a much simplified, indeed
universal approach with which to deal with any kind of lot, material,
situation w.r.t. sampling, defined in the document DS 3077 [2], and
Esbensen and Minkkinen [3].
Due to the fact that the aggregate sum of all (virtual) groups
constitutes the physical lot in its geometric entirety, it follows that
DHL in fact is a measure of the total spatial heterogeneity exhibited by
the lot, hence the term “distributional heterogeneity”, DHL. In this
complementary view, the lot is made up of a set of potential (virtual)
groups, NG in total. This two-scale understanding of the heterogeneity
of any lot, system or material constitutes the effective theoretical
concept with which one is able to understand all key issues of
heterogeneity and representative sampling. DHL accounts for the
material heterogeneity in a specifically relevant form, namely that
corresponding to the specific sampling size used, characterised by a
specific sample mass, MS. The effective group in question is the
sampling increment used. It is possible to ascertain the quantitative
effect of the lot heterogeneity interacting with alternative sampling
processes, for example using alternative sampling volumes or by
using an alternative sampling procedure. This would result in a
numerically different measure of the spatial heterogeneity.
A specific type of increment, termed a grab sample has historically
been very much used, particularly in industrial applications of which
the pharmaceutical industry is one—and the most important aspect
of any such sampling process is the size of this sampling unit MS and
the way it has come about. But as will become clear immediately
below, such a single-scoop sample is very, very rarely acceptable
(results in unacceptably inflated sampling bias), so MS is nearly
always to be understood as the compound mass of a composite
sample, which consists of Q increments. One of the most important
objectives of TOS is to allow the sampler the most reliable way to
estimate the magnitude of Q, refer to DS 3077 [2].
Unlike CHL, which is an intrinsic characteristic of the given
material, DHL can actively be reduced, especially by choosing a
smaller sampling tool, thereby increasing the number of increments in
composite sampling/sampling frequency, and/or the lot can be
thoroughly mixed, blended etc. In large lots, forced mixing is often
impractical or impossible, however; in such cases increasing the
number of increments is the only option for reliable primary sampling.
If there is a significant segregation or grouping (fragment clustering) in
the lot, increasing the sample size for a one-increment sample, MS,
only results in a comparatively minor effect and will soon reach an
impractical limit. TOS has much to say (all negative) regarding the
universal futility of such grab sampling. Grab sampling is in fact never
reliable and must accordingly be abolished.
By way of contrast and effectiveness, composite sampling is
always a good choice of action. It is advantageous to think of more
increments as synonymous with better lot coverage, as illustrated in
Figure 3.1.
It follows that sampling from a heterogeneous lot can never result
in completely identical analytical results; there will always be a
sampling variance (more accurately, a sampling-and-analysis
variance) as expressed by a set of analytical results. Even a set of
identically replicated samples, carried out following an identical
protocol, will give rise to a distinct, non-zero sampling variance (see
section 3.6: Replication experiment). This is solely due to the fact that
no sampling process can fully eliminate the effect of heterogeneity at
all possible scales. The role of representative sampling is to reduce
this fundamental sampling effect as much as possible, and to be able
to quantify the remaining sampling variance. It may happen that
particular systems possess extraordinarily small heterogeneities, but
no generalisations as to universal relationships regarding
“homogeneity” or “sufficient homogeneity” can be drawn from such
particular instances. It is highly advisable always to treat any lot
material as if it carried a significant degree of heterogeneity.
3.3 Sampling error vs practical sampling
TOS’ analysis of the phenomenon of heterogeneity outlines three
factors which are responsible for the magnitude of the distributional
heterogeneity:
CHL (constant for any given material in a specific state).
Grouping (depends on the size (volume/mass) of the extracted
increments).
Segregation (depends on the spatial distribution of
fragments/groups in the lot).
Both segregation and grouping can be quantified if need be;
methods and equations are described in detail in the pertinent
literature and in section 3.9. More important, however, is how to
counteract the effects arising from DHL in practical sampling. In order
to extract samples from heterogeneous materials with sufficiently low
sampling variation it is necessary to minimise the effective DHL. For
any given material state, the case of reducing the two
phenomenological factors grouping and segregation can principally
be achieved in two ways only:
Decreasing the size of the extracted increments, thereby increasing
the number of increments (or increasing the sampling frequency)
combined to form a total sample mass, MS. The purpose of more
increments is to be able to cover the volume of the lot in a more
effective, and hence satisfactory fashion. This approach
counteracts grouping and segregation on the scales between
increment and the whole lot.
Mixing/“homogenising” the lot reduces macro-scale lot
segregation. Mixing is used universally at all sampling stages
except the primary sampling stage where, obviously, this option is
not possible—unless a particular original lot is very small. Thus,
mixing is not a universally available option, whereas composite
sampling always is.
Figure 3.1: “Lot coverage” illustrated in the case of composite sampling from an aggregate
stationary lot. Left, insufficient lot coverage (too small sampling “footprint”). Right, better
coverage, but sampling is still only taking place at the surface of the lot, which is not good
enough—the interior 90% of the lot is structurally never possible to sample (full disclosure
can be found in DS 3077, 2013). See also Figure 3.7.
Thus, all SUOs may, should and will operate on lots at all scales—
from the largest conceivable lot size to the final, pre-aliquot powder
mass immediately before analysis. It is only the physical and
geometrical manifestation of the sampling equipment which
“scales”—in this context a spatula, a scoop, a shovel (spade), a front
loader bucket, are all identical: all are “scoops” taking out
increments. This makes representative sampling completely general
and easy to implement.
Figure 3.7: RSV assessment can be invoked for any sampling procedure/sampling stage.
Here different lot coverage for biological material is testing composite sampling (Q = 11
increments) vs grab sampling employing a singular sample (blue). If a data analysis,
unwittingly or otherwise, is based on analytical data ultimately pertaining to grab samples,
this figure illustrates how spatial heterogeneity is not represented, unless by using composite
sampling. Data analysts should also be interested in the pre-history of their data.
3.9 References
[1] Esbensen, K.H. and Wagner, C. (2014). “Theory of Sampling
(TOS) vs. Measurement Uncertainty (MU)—a call for
integration”, Trends Anal. Chem. 57, 93–106.
https://doi.org/10.1016/j.trac.2014.02.007
[2] DS 3077 (2013). DS 3077. Representative Sampling—Horizontal
Standard. Danish Standards. www.ds.dk
[3] Esbensen, K.H. and Minkkinen, P. (Eds) (2004). “Special Issue:
50 years of Pierre Gy’s Theory of Sampling. Proceedings: First
World Conference on Sampling and Blending (WCSB1).
Tutorials on Sampling: Theory and Practise”, Chemometr. Intell.
Lab. Syst. 74(1), (2004).
[4] Esbensen, K.H. and Julius, L. (2013). “DS 3077 Horizontal—a
new standard for representative sampling. Design, history and
acknowledgements”, NIR news 24(8), 16–19.
https://doi.org/10.1255/nirn.1406
[5] Esbensen, K.H. and Mortensen, P. (2010). “Process Sampling
(Theory of Sampling, TOS) – the Missing Link in Process
Analytical Technology (PAT)”, in Bakeev, K.A. (Ed.), Process
Analytical Technology, 2nd Edn. Wiley, pp. 37–80.
https://doi.org/10.1002/9780470689592.ch3
[6] Petersen, L., Dahl, C. and Esbensen K.H. (2004).
“Representative mass reduction in sampling – a critical survey
of techniques and hardware”, in “Special Issue: 50 years of
Pierre Gy’s Theory of Sampling. Proceedings: First World
Conference on Sampling and Blending (WCSB1)”, Esbensen,
K.H. and Minkkinen, P. (Eds). Chemometr. Intell. Lab. Syst.
74(1), 95–114.
[7] Esbensen, K.H. and Romañach, R.J. (2015). “Proper sampling,
total measurement uncertainty, variographic analysis & fit-for-
purpose acceptance levels for pharmaceutical mixing
monitoring”, in “Proceedings of the 7th International Conference
on Sampling and Blending, June 10-12, Bordeaux”, TOS forum
Issue 5, 25–30. doi: https://doi.org/10.1255/tosf.68
[8] Esbensen, K.H., Román-Ospino, A.D., Sanchez, A. and
Romañach, R.J. (2016). “Adequacy and verifiability of
pharmaceutical mixtures and dose units by variographic
analysis (Theory of Sampling) – A call for a regulatory paradigm
shift”, Int. J. Pharmaceut. 499, 156–174.
https://doi.org/10.1016/j.ijpharm.2015.12.038
[9] Esbensen, K.H. and Julius, L.P. (2009). “Representative
sampling, data quality, validation – a necessary trinity in
chemometrics”, in Comprehensive Chemometrics, Brown, S.,
Tauler, R. and Walczak, R. (Eds). Wiley Major Reference Works,
Vol. 4, pp. 1–20. Wiley, Oxford. https://doi.org/10.1016/b978-
044452701-1.00088-0
* TOS is able to simultaneously consider both original units, “grains”, as well as all possible
fragments hereof produced by the sampling process in operation, by calling both entities
“fragments” (i.e. both true fragments as well as unaffected original grains). This ingenious
conceptual twist allows a comprehensive description of all types of materials—whether they
are but only a little, some or much subjected to fragmentation (or not at all).
Chapter 4: Fundamentals of principal
component analysis (PCA)
4.4.1 Purpose
The data set consists of 17 objects and 6 variables. The objects are
17 cities in Europe and the variables represent yearly consumption
(2012) of various beverages:
Beer consumption litres per year
Figure 4.6: Projections onto a PC; each object has a transverse residual (not all are drawn in
this projection). All projections are perpendicular with respect to the subspace modelled
(here a 1-dimensional PC).
There is an upper limit to the number of PCs that can be derived from
an X-matrix. The largest number of components is either n – 1
(number of objects –1) or p (number of variables), depending on
which is the smaller. For example, if X is a 40 × 2000-dimensional
matrix (40 spectra, each with 2000 variables), the maximum number
of PCs is 39. In this case, the largest number of potential PCs is
limited by the number of samples. Notice that the maximum number
of components is being discussed. In general, it will suffice with a
much smaller number, << p.
From a data analytical point of view, using the maximum number
of PCs corresponds to a simple change of co-ordinate system from
the p-variable space to the new PC space, which is also orthogonal
(and with uncorrelated PC axes).
Mathematically, the effective dimension of the PC space, i.e. the
space spanned by the number of PCs in the model developed (much
smaller than p), is termed the rank of X. The methods used to
estimate this critical parameter of a PC model are discussed below.
Since the first few PCs typically define the major fraction of the
variation in a data set, the later PCs will lie along directions where
there is progressively less and less spread in the objects, or perhaps
there are no longer any significant underlying phenomena to model.
These higher order PCs may only be representing random noise.
Thus, all higher-order directions can progressively be thought of as
potential “noise” directions. This gives an indication of how PCA can
decompose the original data matrix X into a structured part (the first
PCs that span the largest variance directions that represent the major
structured phenomena), and the noise part (directions in the data
swarm where the variance/elongation is small enough perhaps to be
neglected).
where
Figure 4.8: Centring the PC coordinate system at the average data object (translating the
original coordinate system).
Figure 4.11: Example of a 2-D score plot PC1 and PC2 (t1 vs t2) with external information
annotated for more comprehensive interpretation.
From the loading plot it can be seen that other variables also
contribute to PC1, but at the opposite end of the p1 axis (represented
by negative loadings on PC1), these are the variables “Mealiness”,
“Hardness” and “Off Flav”. It does not take an expert to deduce that
sweet peas do not have an off flavour, nor are hard mealy peas fruity
and have a characteristic pea flavour. Therefore, properties that lie at
approximately 180° along a PC direction from each other typically
represent opposite features that are negatively correlated with one
another to different degrees.
Due to the orthogonal properties of PCs, it can be deduced that
Sweetness has very little to do with the property described by PC2.
From the loadings plot, PC2 seems to be describing the colour of the
peas, with those located in the positive PC2 direction being greener
than those in the negative PC2 direction (being whiter in colour).
Some variables display positive loadings, that is to say positive
coefficients in the linear combinations, while others have negative
loadings. For instance, “Sweetness” contributes positively to PC1,
while “Off Flav” has a negative PC1 loading (as well as a very small
positive PC2 loading). PCA loadings are usually normalised to the
interval [–1,1], more of which in the next section on Correlation
Loadings.
Figure 4.13: Correlation loadings of the pea sensory data, PC1 vs PC2. Compare Figure
4.12.
Figure 4.14: Correlation loadings for the pea sensory data, PC3 vs PC4.
Figure 4.15: Comparison of 2-dimensional and 1-dimensional correlation loadings plots for
spectroscopic data.
4.10.2 Comparison of scores and loading plots
Figure 4.16: Complementary scores and loadings plots establishing the optimal basis for
data structure interpretation.
4.11.1 Introduction
Figure 4.18: Line plot of the temperature data, annotated and grouped after region.
A PCA model was performed with full cross validation. The score plot
for the first two PCs is shown in Figure 4.19. The model explains 87%
in the first and 8% in the second PC. The validation variance (not
shown here) was almost identical; 87% and 7%, which indicates that
the model is stable with respect to successively keeping out one city
and project it onto the model based on the other 25. The ellipse
(known as Hotelling’s T2 ellipse) represents a 95% confidence interval
around the model mean and is one criterion for detecting outliers. As
can be seen, there are no clear outliers in the model space when PC1
and PC2 are jointly assessed, however, Moscow lies slightly outside
the ellipse. Based on the position of cities in the score plot, the first
component corresponds to a geographical north–south axis, whereas
the second component describes the complementary east–west axis.
This latter observation is interpreted more to reflect varying proximity
to the Atlantic Ocean, which is the probable causal effect concerning
the yearly temperature profile, and not the geographical east–west
axis per se.
The loadings show to what extent the monthly variables are
important in the various PCs, as a function of how the variables were
scaled before the modelling. It is common to present loadings as line
plots for data where it is most natural to look at the raw data as one
line per object. The first loading in Figure 4.20 confirms that
temperatures are indeed higher in southern Europe than in the
northern parts (positive loadings for all months). Also, the north–south
difference is slightly more pronounced in the winter than in the
summer as was (perhaps) also seen by the data analyst in the line
plot of the raw data.
Figure 4.19: Score plot for PC1 vs PC2 for the European City Temperature data.
Figure 4.20: Line plot of loadings for the first two PCs for the European City Temperature
data set. (Blue line PC1, red line PC2)
Figure 4.22: Influence plot after two PCs of the European City Temperature data set.
Figure 4.23: Score plot of PC3 vs PC1 of the European City Temperature data set.
Figure 4.24: Line plot of loadings for PC3 of the European City Temperature data set.
4.12.2 Centring
The X-matrix used in the equations above is not cast precisely as the
raw data set. The original variables have first been centred using
equation 4.1:
The basis for the step by step calculation of PCs is the Non-Iterative
Partial Alternating Least Squares (NIPALS) algorithm [23], see section
4.14. The fact that T and P are both orthogonal results in a very
efficient algorithm for PC calculation. Suffice here to highlight that it is
the specific delimitation between the data structure part, TPT, and the
noise part, E, which is carried out by establishing the numerical value
of Aopt. When Aopt has been determined (correctly), the PCA model
can be considered complete. Of course, the key question remains:
what does “determined correctly” mean?
A signifies the number of PCs to be calculated; A << p. Recall that
the maximum number of possible PCs is min(n – 1, p). If the number
of PCs calculated is deliberately chosen to be A ≡ min(n – 1, p), a so-
called “full model” has been calculated. This means that
decomposition of X amounts to replacing the original co-ordinate
system (the variable space) with a new co-ordinate system (the PC
space) with the exact same full dimensionality, p. In this case the new
origin corresponds to the mean data object, due to centring, but the
number of PCs (and therefore the dimensionality) has not been
reduced. Only by determining Aopt << p, can the PCA advantage, the
splitting of X into a structure part and a noise part, be achieved. If the
full set of PCs is used, all the noise contributions remain in the
decomposition and therefore E will be 0.
The data analyst, or the software PCA program, must arrive at Aopt
in order that the model, TPT, only contains the “relevant structure” so
that the “noise”, as far as possible, is collected in E. This is a central
theme in most of multivariate data analyses. This objective is not
trivial, however, and there are some pitfalls.
Most PCA software will provide information on which to base a
decision, or will trigger-happily offer a proposal for the “correct model
dimensionality”, Aopt, but in the end it is up to the analyst to decide
how many PCs to use. There are many situations where the human
cognitive facilities excel, which simply cannot be programmed
regardless of how far the field of Artificial Intelligence has been
developed. The software suggestions for optimal A are always
algorithmically derived, and in this endeavour everything hinges on
which optimality criteria is used by the particular “A-algorithm”.
Rather than skipping this important point by decreeing that all new
data analysts can safely rely on such an algorithmic approach, this
book emphatically demands that the readers take it upon themselves
to fully understand the underlying principles behind PCA, so as to be
able to take proper responsibility for this key issue and therefore also
be able to assess the validity of the algorithmic offerings, which are
also routinely spewed out by all competent software packages (“as a
service to the user”).
There are thus no two ways about it. The responsible question
(always) is: “how is the optimum number of PC model components,
Aopt, arrived at”? The answer is in principle simple: keep track of the
magnitude of the E-matrix as the PCA model evolves one PC after
another—and understand full well how to use all three types of PCA
visualisations (scores, loadings, residual variances). Experience is
everything. Do as many exercises as possible—and then some!
and the residual variance is ResVari = /p. Taking the square root of
this sum delivers a measure of the geometric distance between
object i and the model “hyper plane”, i.e. the “flat” or space spanned
by the current A PCs as expressed in the original variable space.
Thus, the object residual is nothing but the geometric projection
distance between the object and the model representation. The
smaller this distance is, the closer the PC representation (the “foot
point”) of the object is to the original object. In other words, the rows
in E are directly related to how well the model fits the original data—
by using the current number of A model components.
Figure 4.26: Relative residual variances for the peas sensory data.
The details in the residual variance plot, Figure 4.27, help find the Aopt
for a given data set. There are several ways to go about this.
There exists an empirical rule which says that a large break in this
function—going from a steep slope to a much slower decrease—
often represents the “effective” dimensionality; the optimum A. In
Figure 4.27 this is at PC number 1 (with 3 representing the point
where there is no more variation). There is a logical argument behind
this rule of thumb. Recall that the PCs represent directions of
maximum variances, i.e. along the elongated directions of the data
swarm, in decreasing order. When projected down onto these
directions, the distance measure from the objects to the PC model
will decrease. Also, remember the duality of maximum elongation
variances and minimisation of the transverse residual distances for
each component. As long as there are (still) new directions in the data
swarm with relatively “large” variances, the total residual distance
measure will decrease significantly when adding still one more PC.
This translates into a relatively large decrease in the total residual
variance from PCa to PCa + 1, corresponding to a relatively steep slope
in the plot, from component a to the next.
One may use the mental metaphor that the next PC would still
have something “to bite into”, or that there still is some definite
direction in the remaining projections of the data swarm for the next
component to model. This goes on until the remaining data swarm
does not show preferred orientations (elongations) any longer. At this
point there will no longer be any marked gain with respect to adding
any further PCs (“gain” is here to be understood as modelling gain,
represented by a significant total variance reduction). Consequently,
the total residual variance curve will start to flatten out and the gain
per additional PC will be significantly less than before, thus producing
a slope break. Once this “noise region” has been reached, all of the
subsequent PCs will represent slowly decreasing E matrices as the
added components are in fact only modelling random variation.
To conclude: the optimal number of PCs to include often is the
number of PCs that gives the clearest break point in the total residual
variance plot. But this is only a first rule-of-thumb. Alas, there are
plenty of exceptions from this rule most of which are because of one
form or other of “special” data structure. Note that non-standard
variance plots will be the rule, for example, as long as influential
outliers have not yet been policed out of the training data set. Marked
“clumpiness” is also a special data set structure that will give rise to
an apparent number of components that very likely will not stand up
to diligent final modelling. This issue is taken up again in the central
chapter on validation (chapter 8).
Here only the former will be discussed. The residual variance per
variable can be used to identify non-typical variables, “outlying”
variables, in a somewhat similar fashion as for the objects. These
cannot, however, be interpreted in an exact analogous fashion in
terms of distances without introducing a complementary object
space in which variables can be plotted. This concept lies outside the
scope of this introductory book, however.
4.13.1 Purpose
4.13.4 How to do it
Depending on the qualifications of the sensory panel, it is common
practice to auto-scale the data (refer to chapter 5). The present pea
model was auto-scaled data and the method of cross validation was
used to validate the model (see chapter 8 re. the necessary
qualifications).
The PCA overview for this model is presented in Figure 4.29. It
can be seen from the Explained Variance plot that the first two PCs
explain around 79% of the total variance in the data. This is regarded
as good for sensory analysis, due to the relatively high noise level
(relatively high sensory judgment reproducibility variance) in this kind
of “measurement”.
The Explained Variance plot suggests that there are three or four
components that could be interpreted (note the point where the blue
curve flattens out in the plot). The red curve is the validation curve
and is an indication of model quality. More on this in chapter 8.
Now study the score plot. For convenience, it has been grouped
by Harvest Time (as previously shown in Figure 4.11). Previously is
was stated that the PC1 direction was influenced by time (growing,
maturing), which is directly related to the variables “Sweet”, “Fruity”
and “Pea Flavour”. These are diametrically opposed to the variables
“Mealiness”, “Hardness” and “Off Flavour”.
To understand the nature of the second source of variation, along
PC2, the variables “Colour 1”, “Colour 2”, “Colour 3” and
“Whiteness” occupy the PC2 space. “Whiteness” and “Colour 1”
occupy a similar space and are therefore correlated. This would
indicate that “Colour 1” and “Whiteness” are a measure of the same
thing. At approximately 180° to these variables, the variables “Colour
2” and “Colour 3” lie (imaging a line connecting the two sets of
diametrically opposed variables passing through the origin). This
indicates that “Colour 2” and “Colour 3” are correlated and since
green peas are being assessed, it can be concluded that these two
variables are a measure of the greenness of the peas. This indicates
that PC2 is a “colour direction”. It does not line up exactly with PC2
and therefore, colour and flavour (PC1) are also correlated. This is
why the scores look like they distribute along a gradient along PC1
and PC2.
The scores and loadings pairs for PCs 3 and 4 are provided in
Figure 4.30, this time displaying the loadings as correlation loadings.
From Figure 4.30, it can be seen that PC3 describes a further 10%
of the total data variability and is mainly only related to the variable
“Texture”. The scores plot indicates no trending of texture to Harvest
Time or Cultivar. PC3 can therefore be assigned as a measure of pea
texture and nothing else. PC4 describes an additional 5% of the data
variability and is in turn weakly related to the variable “Skin”. The
correlation loadings plot would indicate that this variable is not highly
correlated to PC4, however, and a conclusion could be not to include
PC4 as it is not really informative, it is only indicative.
Figure 4.29: PCA overview of the assessment of green peas using sensory data.
4.13.5 Summary
There is no clear break in the Explained Variance plot, but two PCs
describe 79% of the total variance while the third explains an
additional 10%. The two first PCs are simple to interpret and are
probably sufficient to determine the most important variables for
description of pea quality. The clue here is to carefully follow the
quantitative fractions of the total variance associated with each PC.
The score plot showed that the PC1 direction describes the
harvesting time since the samples are distributed from left to right
according to their Harvesting Time numbering. There is no similar
obvious clear pattern in PC2 from the scores plot.
The loading plot shows that “Pea Flavour”, “Fruitiness” and
“Sweetness” co-vary. “Hardness”, “Mealiness” and “Off Flavour” are
also positively correlated to each other, while they are negatively
correlated to “Pea Flavour”, “Sweetness” and “Fruitiness”, since the
two groups are on opposite sides of the origin along PC1 (i.e. they are
diametrically opposed). This means that PC1 mostly describes how
the peas taste and feel in the mouth, which is perhaps not so
surprising: the first direction is defining what trained sensory judges
base their assessment on. The corresponding score plot indicates
that taste is indeed related to harvest time—the riper the peas, the
sweeter they taste.
Figure 4.30: PC scores and loadings for PCs 3 and 4 for the assessment of green pea
quality.
4.16 References
[1] Fisher, R.A. (1936). “The use of multiple measurements in
taxonomic problems”, Ann. Eugenics 7, 179–188.
https://doi.org/10.1111/j.1469-1809.1936.tb02137.x
[2] Geladi P. and Esbensen, K. (1990). “The start and early history
of chemometrics: Selected interviews. Part 1”, J. Chemometr. 4,
337–354. https://doi.org/10.1002/cem.1180040503
[3] Esbensen, K. and Geladi, P. (1990). “The start and early history
of chemometrics: Selected interviews. Part 2”, J. Chemometr. 4,
389–412. https://doi.org/10.1002/cem.1180040604
[4] Wold, S., Albano, C., Dunn III, W.J., Edlund, O., Esbensen, K.,
Geladi, P., Hellberg, S., Johansen, E., Lindberg W. and
Schöström, M. (1984). “Multivariate data analysis in chemistry”,
in Chemometrics, Mathematics and Statistics in Chemistry, Ed
by Kowalski, B.R. D. Reidel Publ., pp. 17–195.
https://doi.org/10.1007/978-94-017-1026-8_2
[5] Kowalski, B.R. (Ed) (1984). Chemometrics, Mathematics and
Statistics in Chemistry. D. Reidel Publ. ISBN 978-94-017-1026-
8
[6] Sharaf, M.A., Illman, D.L. and Kowalski, B.R. (1986).
“Chemometrics”, Volume 82 in Chemical Analysis. John Wiley
and Sons.
[7] Esbensen, K.H. and Geladi, P. (2009). “Principal Component
Analysis (PCA): Concept, geometrical interpretation,
mathematical background, algorithms, history”, in
Comprehensive Chemometrics, Ed by Brown, S., Tauler, R. and
Walczak, R. Wiley Major Reference Works, Vol. 4, pp. 211–226.
https://doi.org/10.1016/B978-044452701-1.00043-0
[8] Ehrenburg, A.S.C. (1978). Data Reduction—Analysing and
Interpreting Statistical Data. Wiley. ISBN 0-471-23398-6.
[9] Brereton, R. (1990). Chemometrics, Applications of Mathematics
and Statistics to Laboratory Systems. Ellis Horwood. ISBN 0-13-
131350-9.
[10] Jackson, J.E. (1991). A User’s Guide to Principal Components.
Wiley. https://doi.org/10.1002/0471725331
[11] Johnson, R.A. and Wichern D.W. (1988). Applied Multivariate
Statistical Analysis. Prentice-Hall.
https://doi.org/10.2307/2531616
[12] Krzanowski, W.J. (1988). Principles of Multivariate Analysis—A
User’s Perspective. Oxford Science Publications, Oxford
Statistical Science series No. 3. ISBN 0-19-852230-4.
[13] Mardia, K.V., Kent, J.T. and Bibby, J.M. (1979). Multivariate
Analysis. Academic Press. ISBN 0-12-471252-5
[14] Martens, H. and Næs, T. (1989). Multivariate Calibration. Wiley.
ISBN 0-471-90979-3
[15] Massart, P.L., Vandegiste, B.G.M., Deming, S.N., Michotte, Y.
and Kaufman, L. (1988). Chemometrics: A Text Book. Elsevier.
ISBN 0-444-42660
[16] Höskuldsson, A. (1996). Prediction Methods in Science and
Technology, Vol. 1. Basic Theory. Thor Publishing, Denmark.
ISBN 87-985941-0-9
[17] Jolliffe, I.T. (1986). Principal Component Analysis. Springer
Verlag, New York. https://doi.org/10.1007/978-1-4757-1904-8
[18] Hotelling, H. (1933). “Analysis of a complex of statistical
variables into principal components”, J. Educ. Psychol. 24, 417–
441. https://doi.org/10.1037/h0071325
[19] Pearson, K. (1901). “On lines and planes of closest fit to
systems of points in space”, Phil. Mag. Ser. B. 2, 559–572.
https://doi.org/10.1080/14786440109462720
[20] Harmon, H.H. (1976). Modern Factor Analysis, 3rd Edn.
University of Chicago Press.
[21] Lorho, G., Westad, F. and Bro, R. (2006). “Generalized
correlation loadings. Extending correlation loadings to
congruence and to multi-way models”, Chemometr. Intell. Lab.
Syst. 84, 119–125.
https://doi.org/10.1016/j.chemolab.2006.04.023
[22] Swarbrick, B. and Westad, F. (2016). “An overview of
chemometrics for the engineering and measurement sciences”,
in Handbook of Measurement in Science and Engineering, Ed.
by Kutz, John Wiley & Sons, Hoboken, NJ, pp. 2331.
https://doi.org/10.1002/9781119244752.ch65
[23] Wold, H. (1966). “Estimation of principal components and
related models by iterative least squares”, in Multivariate
Analysis, Ed by Krishnaiah, P.R. Academic Press, NY.
[24] Geladi, P. (1988). “Notes on the history and the nature of partial
least squares (PLS) modelling”, J. Chemometr. 2, 231–246.
https://doi.org/10.1002/cem.1180020403
* Elsewhere in this book (see chapter 5 on scaling and pre-preprocessing), it is discussed
whether, and why, it is advisable to apply auto-scaling by default. It will be appreciated that
this issue more revolves around whether the empirical variance is approximately constant
over the entire wavelength range, than with a specific “type” of data.
Chapter 5: Preprocessing
5.1 Introduction
It is a fair statement that in more cases than not, data requires some
form of preprocessing. In general, preprocessing is but a minor
modification of a data set, for example, to minimise the impact from
“extraneous noise” such that structural information is more readily
extracted. An excellent analogy is viewing the moon on a slightly
cloudy night. It is obvious that the moon is in the sky but the thin
layer of cloud does indeed distort from a desired clear view. If
preprocessing could be likened to a strong wind, this could blow the
layer of cloud away, making the moon much more visible and clear.
There are three major data types when preprocessing methods
are considered. These are:
1) Discrete variables. These originate from systems that generate
process data such as temperature, pressure, pH readings etc. or,
for instance, sensory data where individual products are assessed
for characteristics such as taste, colour etc.
2) Spectroscopic variables. These are typically the most common
variables encountered in chemometrics. These are typically
absorbance values generated at each wavelength (wavenumber) for
methods such as infrared, near infrared, Raman, UV-vis. There are
other types of spectroscopic-like variables, e.g. acoustic frequency
variables.
3) Time Series variables, These are typically of the form of response
variables measured over time at regular intervals and various
methods exist to extract the true underlying structures of noisy
data.
A new generation of models that includes aggregation (“fusion”) of
discrete variables with spectral data are also available that require
methods such as block weighting of the data. For the purposes of
this chapter, only discrete and spectroscopic variable preprocessing
will be discussed. The interested reader is referred to the text by
Montgomery, Jennings and Kulahci [1] for a more detailed discussion
of how to preprocess time series data.
Chromatographic data can fall into a grey area between
discrete/spectroscopic and time series due to the nature of the data
and its generation. This chapter briefly touches on the preprocessing
of chromatographic data as well.
As mentioned, preprocessing is a method of minimising
extraneous noise in a data set. This is important for the multivariate
methods discussed in chapters 4, 7 and 10, since these methods aim
to extract the greatest sources of variability in the data set. If noise
effects are large in the data that can easily be minimised by some
form of simple mathematical transformation, then the effect is
explicitly removed before the model is developed, rather than making
the model more complex by implicitly removing the effect using a
model factor.
7 ≤ pH ≤ 7.5
250 ≤ Temperature (°C) ≤ 260
1200 ≤ Pressure (kPa) ≤ 1250
5.2.3 Averaging
Row-averaging is used when the goal is to reduce the number of
samples in a data set, for example to reduce the effect from
measurement uncertainty (e.g. from reference measurements or
sampling) or to reduce the effect of noise (e.g. instrumentation
precision limitations). Data sets with many replicates of each sample
can also often be averaged with the advantage to ease handling
regarding validation and to facilitate interpretation. The result of
variable averaging (or column-averaging) is a smoother data set. But
there is a price to be paid in reduced resolution; averaging is by no
means always a beneficial option—everything depends on the data
analysis context.
5.3.2 Smoothing
Savitzky–Golay smoothing
The Savitzky–Golay filter (also available as a derivative function, refer
to section 5.3.5) fits a low order polynomial function to the window
size defined. In this case the centre point of the window is replaced
by an estimate of the fitted polynomial and like the moving block
smoothing algorithm, it is applied recursively across the region of
interest. It is likewise a requirement of the Savitzky–Golay filter
window that the window size is an odd integer value. Figure 5.7
shows how the Savitzky–Golay smoothing filter works and more
details are provided when a discussion of the derivatives is presented
later in this chapter.
5.3.3 Normalisation
Normalisation is a row wise (or object wise) method that puts all
objects (or variables) on an even footing. Many of the transformations
described and applied so far have been column-transformations, i.e.
making specific preprocessing or transformations which act on one
column-vector individually (single-variable transformations). The
process of normalisation rescales, i.e. normalises, each object into a
common sum, for example 1.00 or 100%.
The row sum of all variable elements is computed for each object
individually. Each variable element is then divided by this object sum.
The result is that all objects now display a common size—they have
become “normalised” to the same sum area in this case.
Normalisation is a row analogy to variance scaling (refer to section
5.2).
Normalisation is a common object transformation. For instance, in
chromatography it is used to compensate for (smaller or larger)
variations in the amount of analyte injected onto the chromatographic
column. It would be of considerable help in the analytical process if
this particular measurement variance can be controlled by a simple
data analytic preprocessing, otherwise (as will be discussed in great
detail in chapter 7), the inclusion of an increased number of data
model components would have to be included in order to model such
irrelevant input variations. In such cases, the use of preprocessing
explicitly removes unwanted physical artefacts rather than relying on
modelling to implicitly account for them, thus unnecessarily
increasing its complexity.
Figure 5.7: Savitzky–Golay filter applied to a noisy Gaussian peak.
where Xuncorr is the uncorrected spectra, Xtrue is the true signal free of
baseline effects and α + βx are to be determined to correct the
spectra such that,
5.3.5 Derivatives
Figure 5.13: First derivative spectra of the baseline and linearly sloping Gaussian
chromatographic data.
The first derivatised data shows how the baseline and slope effect
are simultaneously removed. The peak maxima in the original data is
now shown as a zero point. This is because the slope at the peak
maxima is zero. This is sometimes why first derivatives are not the
preferred method for the analysis of spectroscopic data as zero
points are often difficult to interpret when the spectra become
complex. In these cases, the second derivative is preferred as shall
be explained in the following. Returning to the quadratic equation
introduced earlier in this section, the second derivative is provided as
follows,
Figure 5.14: Second derivative spectra of the baseline and linearly sloping Gaussian
chromatographic data.
Comparison of derivatives
Derivative orders
The reason for the presence of such noise can be explained from
a mathematical point of view. The original data were collected on an
absorbance scale between 0 and 1. Data point resolution determines
the vertical distance between two successive absorbance values. For
the spectrum in Figure 5.16, the difference between two absorbance
values for a major peak is in the order of 0.007 absorbance units. This
is dangerously close to the signal to noise limit of the instrument for
which reason derivatisation can in fact amplify noise. The noise is
further amplified for the second derivative and is shown for
comparison for the FT-NIR data in Figure 5.17.
Figure 5.16: Simple first difference FT-NIR spectrum showing how noise can be amplified in
the derivative calculation.
Note the scale differences between the raw data, the first
derivative and the second derivative spectra. In each case, there is an
approximate two order of magnitude decrease in the scale and for
the second derivative, this is now bordering on the signal to noise
capabilities of the instrument. For a detailed discussion of signal to
noise ratios in analytical instrumentation, the interested reader is
referred to the book by Adams [7].
To overcome such issues with noisy derivatives, two commonly
used derivative algorithms are
1) The segment-gap derivative and
2) The Savitzky–Golay derivative
Figure 5.17: Simple second difference FT-NIR spectrum showing how noise can be amplified
in the derivative calculation.
Full MSC:
Figure 5.21: Application of MSC to NIR spectra collected in the 1100–2500 nm region of the
spectrum.
In this model, the first two terms are the usual MSC model. The
supplementary terms and represent “channel weighting
factors” to take into account linear and quadratic wavelength
dependent effects. The use of EMSC will be shown in section 5.4.2
where this time a series of chemically diverse spectra will be used as
an example.
An extension to the EMSC model is known as modified EMSC
(mEMSC) first introduced by Martens et al. [13]. In this method, the
EMSC equation is extended with a term to include external
information to direct the scatter effect preprocessing. The new
information (defined by the term G in equation 5.15) is often supplied
in the form a “good spectrum”, i.e. a known constituent’s pristine
information can be used to import useful information in the
preprocessed data—or it can actually be in the form of a “bad
spectrum”, i.e. an interferent whose influence of the spectrum is to be
removed as much as possible. These features make the mEMSC
approach very versatile. The use of the mEMSC model will also be
demonstrated by example in section 5.4.2.
Detrending
In their original paper on SNV, Barnes et al. [10] suggest that after
preprocessing the data with SNV, this should be followed by the
process of detrending. Detrending is a transformation which
minimises non-linear trends, thus SNV and detrend in combination
reduces multicollinearity, baseline shift and curvature. Detrending is
similar to a sloping baseline correction (refer to section 5.3.5) and
calculates a baseline function as a least squares fit of a polynomial to
the sample spectrum. These transformations are applied to individual
spectra and are distinct from other transformations which operate at
each wavelength in a given set of spectra. As the polynomial order of
the detrend increases, additional baseline effects are removed. (0-
order: offset; first-order: offset and slope; second-order: offset, slope
and curvature.)
Typically, detrending is performed by using a second-order (or
higher degree) polynomial in regression analysis where absorbance
values (or y-variables) and the independent variable or x-variable (w)
is given by the corresponding wavelengths (equation 5.16):
COW, Tomasi [14], is a method for aligning data where the signals
exhibit shifts in their position along the x-axis. COW can be used to
eliminate shift-related artefacts in measurement data by correcting a
sample vector to a reference. COW has applicability to data where
there can be a poor alignment of the x-axis from sample to sample,
as can be the case for example with chromatographic data, Raman
spectra and NMR spectra. One example of such data is
chromatography, where peak positions change between samples due
to changes in mobile phase or deterioration of the column. Another
example is in NMR spectroscopy, where matrix effects and the
chemistry itself induce position changes in the chemical shifts.
The method works by finding the optimal correlation between
defined segments of the data for which there is a shift in position. The
result of this procedure is one shift value per segment. These are then
interpolated to give a so-called shift-vector for all data points, and a
mapping function (move-back operator) which moves the samples
back to the reference profile’s position.
To cope with various shift lengths, it is suggested to pad a data
table with zeros before performing the shift alignment. Alignment is
done by allowing small changes in the segment length on the sample
vector, and those segment lengths being shifted (“warped”) to
optimise the correlation between the sample and the reference
vector. Slack refers to the maximum increase or decrease in sample
segment length, and provides flexibility in optimising the correlation
between the samples and reference.
One of the simplest and most powerful tools a data analyst has at
their disposal for determining the type of preprocessing to use on a
data set is the scatter effects plot. This plot shows each sample
plotted against the average (mean) sample for a selected set of
samples (the object base). Scatter effects display themselves as
differences in slope and/or offset between the lines in the plot.
Differences in the slope are caused by multiplicative scatter effects.
Offset error is due to additive effects. Sometimes the lines show
profiles that deviate considerably from a straight line. In such
instances, caution must be taken when applying scatter correction,
as major chemical information may be confused with systematic
scatter effects and therefore lost in the transformation. For an
excellent reference of this situation, refer to the article by Martens et
al. [13] and section 5.4.2 where the gluten–starch data set is
presented.
Applying multiplicative scatter correction may improve the model
if these types of scatter effects are observable, but as stated over
and over, caution must always be exercised when applying any
preprocessing method to a data set. Figure 5.24 provides four
example situations to look for when assessing the scatter effects plot.
The four situations described in Figure 5.24 are described as
follows:
Situation a): When the individual spectra in a set have exactly the
same profile as the mean spectrum, plotting the data against the
mean will result in straight line fits. The reason the lines do not overlay
each other is due to a purely additive effect. In this case, simple
baseline correction or derivatives should be considered.
Situation b): When the individual spectra form straight lines when
plotted against the mean, again, this indicates that the mean
spectrum captures the principal characteristics of all the samples,
however, the individual spectra have different slopes with respect to
the mean. This is an indication that a scatter effect needs to be
corrected in the data. Since the data emanate from a single point,
there is no indication of an additive effect. In this case methods such
as SNV or common amplification MSC may be useful in correcting
such data.
The data described here come from a publication from Martens et al.
[13] on the use of transmission NIRS for measuring binary mixtures of
gluten and starch at various concentrations. This data is also
presented in chapter 7 on regression to show how the choice of
effective preprocessing can lead to the simplest modelling situations.
Figure 5.25: Scatter plot of FT-NIR data only exhibiting additive effects.
The premise behind this example was that no matter what the
sample packing density or the pathlength used (within reason) for
measuring a binary mixture of powders, the composition of the
powders should be able to be predicted using the simplest (i.e. 1
component/factor model), provided of course that the powder
samples have been carefully prepared and that the area presented to
the analytical instrument is as “homogeneous” as possible for each
sample.
100 spectra of gluten–starch mixtures were collected ranging from
0% to 100% of each constituent in 20% increments. For each
concentration, the powders were scanned in transmission mode in a
variable pathlength cell to introduce variations in the scanning and
the powders were also either packed loosely or tightly to introduce
packing density effects. The spectra of the samples are shown in
Figure 5.26.
Inspection of the data in Figure 5.26 reveals that additive effects
are dominant, i.e. the spectra are highly offset from each other along
the absorbance scale and the other feature observed was an
appreciable chemical diversity between many of the spectra. The
scatter effects plot for this data is provided in Figure 5.27.
In Figure 5.27, the top plot shows all data plotted in the scatter
effects plot where the distinctive offset can be seen. The bottom plot
extracted three of the most diverse spectra in the set, the
characteristic non-linear behaviour indicates that the sample spectra
are very dissimilar from the mean. A scatter correction method like
MSC would not be able to distinguish this variability from scatter
effects and will distort the data structure. This is shown in Figure
5.28.
Figure 5.27: Scatter effects plot of gluten–starch mixtures measured by transmission NIRS.
Figure 5.28: MSC preprocessed gluten–starch data.
The data in Figure 5.28 show that MSC cannot handle the
variability associated with the 100% gluten samples. The
concentration changes between 0 and 100 are also non-linear
indicating that MSC is not an appropriate preprocessing method for
this data. The EMSC algorithm is better able to handle these
wavelength dependent scattering and diversity effects and the
application of this preprocessing method to the data is shown in
Figure 5.29.
The EMSC preprocessing is better able to handle the diversity of
the spectra and shows tight groupings of all of the concentration
ranges. A careful inspection of the absorbance centred around 840
nm shows that gluten concentration varies non-linearly with
absorbance. This was also observed when this data was modelled
using PLSR (refer to chapter 7, section 7.13.6).
Figure 5.31: Specific removal of unnecessary errors in analysis and in data analysis.
5.6 References
[1] Montgomery, D.C., Jennings, C.L. and Kulachi, M. (2008).
Introduction to Time Series Analysis and Forecasting. John
Wiley & Sons.
[2] Montgomery, D.C. (2005). Design and Analysis of Experiments,
6th Edn. John Wiley & Sons.
[3] Chou, Y.M., Polansky, A.M. and Mason, R.L. (1998).
“Transforming non-normal data to normality in statistical
process control”, J. Qual. Technol. 30(2), 133–141.
[4] Beebe, K.R., Pell, R.J. and Seasholtz, M.R. (1998).
Chemometrics: A Practical Guide. John Wiley & Sons.
[5] Deming, S.N., Palasota, J.A. and Nocerino J.M. (1993). “The
geometry of multivariate object preprocessing”, J. Chemometr.
7, 393–425. https://doi.org/10.1002/cem.1180070506
[6] Swarbrick, B. (2016). “Near infrared (NIR) spectroscopy and its
role in scientific and engineering applications”, in Handbook of
Measurement in Science and Engineering, Vol. 3, Ed by Kutz, M.
John Wiley & Sons, pp. 2583–2608.
https://doi.org/10.1002/9781119244752.ch71
[7] Adams, M.J. (1995). Chemometrics in Analytical Spectroscopy.
The Royal Society of Chemistry.
[8] Norris, K. (2001). “Applying Norris derivatives. Understanding
and correcting the factors which affect diffuse transmittance
spectra”, NIR News 12(3), 6. https://doi.org/10.1255/nirn.613
[9] Savitzky, A. and Golay, M.J.E. (1964). “Smoothing and
differentiation of data by simplified least squares procedures”,
Anal. Chem. 36, 1627–1639.
https://doi.org/10.1021/ac60214a047
[10] Barnes, R.J., Dhanoa, M.S. and Lister, S.J. (1989). “Standard
normal variate transformation and de-trending of near-infrared
diffuse reflectance spectra”, Appl. Spectroc. 43, 772–777.
https://doi.org/10.1366/0003702894202201
[11] Martens, H., Jensen, S.Å. and Geladi, P. (1983). “Multivariate
linearity transformation for near-infrared reflectance
spectrometry”, in Proc. Nordic. Symp. Appl. Stat. Stockland
Forlag Publ., Stavanger, Norway, pp. 205–234.
[12] Martens, H. and Stark, E. (1991). “Extended multiplicative signal
correction and spectral interference subtraction: new
preprocessing methods for near infrared spectroscopy”, J.
Pharm. Biomed. Anal. 9, 625–635.
https://doi.org/10.1016/0731-7085(91)80188-F
[13] Martens, H., Nielsen, J.P. and Engelsen, S.B. (2003). “Light
scattering and light absorbance separated by extended
multiplicative signal correction. Application to near-infrared
transmission analysis of powder mixtures”, Anal. Chem. 75,
394–404. https://doi.org/10.1021/ac020194w
[14] Tomasi, G., van den Berg, F. and Andersson, C. (2004).
“Correlation optimized warping and dynamic time warping as
preprocessing methods for chromatographic data”, J.
Chemometr. 18, 231–241. https://doi.org/10.1002/cem.859
6. Principal Component Analysis
(PCA)—in practice
Step 2: Plotting Raw Data: Getting a “Feel” for the Particular Data Set
There is a saying in multivariate analysis, “Visualise before you
analyse”. When dealing with discrete data sets, use bar plots or line
plots to understand the data structure from an initial univariate
perspective. Histograms of single variables can provide valuable
information on data distribution to assess whether variables
potentially contain information, but of course this process becomes
tedious when the number of variables is even only moderately large.
The use of Descriptive Statistics (chapter 2) may be a better option in
these cases.
For spectroscopic or chromatographic data, the use of line plots
is recommended as the general profile of the data can be assessed
and any gross outliers can very easily be detected (and justifiably
removed) before performing multivariate analysis.
Figure 6.2 shows a (t1 vs t2)-score plot for the analysis of gasoline
spectra used to build a calibration model for the prediction of Octane
Number (refer to chapter 7 for more details on this problem), where
three distinct groups can be observed. In this type of situation, it is
advisable to use coded object-names in a way that reflects their
compositions or other known relevant external properties as shown in
this figure. In this case the objects are grouped by high, medium and
low octane number grading.
Necessary Advice: It is the authors’ experience that when
external bodies provide “well-labelled” data sets that it is never a
surprise to find some of the worst labelling practices in play. The best
advice that can be offered is to be absolutely consistent with sample
naming and labelling—double-checking is a bliss. It will make the
lives of all involved much easier and will typically (from experience)
halve the time taken to analyse the data. Too much time is wasted in
data analysis trying to sort out what may be meant by the person who
compiled the data in the first place. Always decide on a consistent
naming convention and stick to it.
To look for patterns, it might be useful to draw lines between
objects with identical properties and to encircle groups of objects
with similar properties, as has been done in Figure 6.2, manually by
all means on a print out, or aided by some relevant computer
software. It is perfectly admissible to use hand-drawn guide lines on
any type of plot—in fact this is recommended throughout the
modelling phase when new information is still emerging and is being
assessed. The issue here is not how to do this—the issue is to use
whatever appropriate annotation to the plot in question, which will
help with the particular interpretation needs.
The annotated plot in Figure 6.2 shows that octane number
increases along a (virtual) axis that runs from lower left to upper right;
i.e. both PCs 1 and 2 contribute to the determination of octane
number and thus in the decomposed score-plot (which above had
been claimed to result in orthogonal, individually interpretable
components). It is of no great surprise to state that real-world
systems are regrettably not always that simple. There is no guarantee
that all data systems will necessarily be structured in such a simple
fashion so as to be stringently decomposable only into one-to-one
phenomena–PC-component relationships. For more complex data
sets, PCA results are nevertheless decomposed into easily
interpretable axes, which require subject matter expertise in the
particular domain to interpret.
As is found in Figure 6.2, software packages such as The
Unscrambler® always list the explained (modelled) calibration
variance for the PCs displayed. In this case, PC1 describes 86% of
the data and PC2 describes a further 12% for a total of 98%
explained in two PCs. Higher order PCs therefore influence less than
2% of the model, so interpretation or outliers in higher order PCs
would be absolutely irrelevant.
Score Plots—Outliers
Figure 6.3 is the scores plot from the same set of gasoline samples
described in Figure 6.2, this time with two samples added that
contain an additive. The scores plot now shows that two severe
outliers, objects marked as the additive, are now detected. In this
case one can appreciate the enormous effect outliers can have on a
PC model. PC1 is used almost exclusively to describe these two
objects in opposition to all others (88% in total). The other objects lie
nicely on a line that, although somewhat oblique to, is well described
by PC2 (10% explained variance). If the objects that are detected as
outliers are erroneous or non-representative (note: they may not be—
even if they at first look very different from the others), they distort the
whole model because of their large leverage effect. If/when they are
removed, the other samples/objects become evenly distributed
throughout the whole score plot (Figure 6.2). Note also that in Figure
6.2 what PC1 described as important accounted for 86% of the total
variability and this information in only 10% explained in PC2 from
Figure 6.3. Such is the effect of large, strongly influential outliers.
Clusters or groups of objects do not always imply problems, but if
such groups are clearly separated, it might be necessary to model
each group separately. Decisions on this type of issue is, needless to
say, only possible based on critical external knowledge. In general, in
score plots, objects close to each other are similar and the further
objects separate, the more dissimilar they become. The topic of
outliers is presented in section 6.6 of this chapter.
Figure 6.2: Annotated score plot. Use problem-specific, meaningful names to ease
interpretation.
Figure 6.3: Score plot with objects with additives revealed as gross outliers.
Correlation / Covariance
Variables close to each other, situated towards the periphery of the
loading plots, co-vary strongly, proportionally to the degree distanced
from the PC origin (again relative to the overall total variance fractions
explained by the pertinent components). If the variables lie on the
same side of the origin, they co-vary in a positive sense, i.e. they
have a positive correlation (auto-scaled data). If they lie on opposite
sides of the origin, more or less (some latitude is needed here) along
a straight line through the PC origin, they are negatively correlated
with each other. Correlation is not automatically reflecting a causal
relation, however; interpretation is always necessary.
Most importantly: loadings, which are at approximately 90° to
each other with respect to the origin, are not co-varying, i.e. they vary
independently of each other. Loadings close to a PC axis are
significant only to that PC. Variables with a large loading on two PCs
are significant on both PCs. Figure 6.4 provides a graphical
representation of these situations.
Figure 6.4: Graphical representation on how to interpret 2D loading structures.
Spectroscopic data
In spectroscopic applications, and similar for many X-variable data
sets, the one-vector loading plot is often the more useful. Again, large
loading values imply important variables (e.g. wavelengths). With
reference to the loadings plot of Figure 6.5, the following conclusions
can be drawn:
Figure 6.5: Spectral line loadings of the PCA performed on gasoline samples by NIR
spectroscopy.
Figure 6.8: PCA scores and explained variance plots for raw spectra alcohol mixture data.
Figure 6.10 shows the score plot with its associated explained
variability plot when applied to the MSC-corrected transmission
spectra. In this case, a two-PC model is an adequate representation
of the data, particularly for mixture data where it is assumed that if
two components of a three-component system are known, the third
component can be found by difference.
The shape of the score plot now resembles the original mixture
design and is an indication of the adequacy of the preprocessing
applied to the data. However, because the points do not align
perfectly with the edges of the simplex, this is an indication of
uncorrected non-linear mixing in the system.
There are plenty of preprocessing methods to try on a data set,
however, as explained in great detail in chapter 5, it is bad form to try
out all alternative scaling, transformations or normalisations
indiscriminately without problem-specific justification. This situation
was exemplified by the previous example where the preprocessing
was chosen as a non-linear correction function and its use was
justified based on the two-dimensional score plot describing 97% of
the total data variability. Preprocessing is not an option that one
selects willy-nilly from shelf abundance in the data analytical
supermarket. At this point, all that is required to know is that the
correct choice of preprocessings are all problem dependent, and that
a wrong choice may actually lead to interpretations that are not
relevant to the specific problem.
Figure 6.10: PCA scores and explained variance plots for MSC-corrected alcohol mixture
data.
This was defined in path 3 in section 6.1 and can to some extent be
observed when any interpretations do not make sense. However,
there are also some very subtle mistakes that can be made here.
Take the case of carrying out a PCA on a data set and the conclusion
drawn is that PC1 corresponds to viscosity, say, while in reality it may
just as well correspond to temperature (but not enough time was
taken to reflect on the problem formulation, and this important
external influence “missed” specifying that temperature should also
be measured). In this case, it is not possible to discover this without
being aware of the temperature dependence of viscosity and defining
the experimental protocol accordingly, for instance, by keeping the
temperature constant in some of the experiments. In a direct PCA,
this kind of lack of reflection of the entire problem domain can easily
lead to misinterpretations, in multivariate regression it can be a
severe problem. Again, it all boils down to subject matter expertise
related to the specific domain of the problem.
6.5.4 Outliers which are truly due to erroneous data were not
removed
All relevant score plots were not investigated carefully, and too many
important clues were missed. Errors 6.5.4 and 6.5.5 above are
connected to this mistake.
6.6 Outliers
Outliers are atypical objects (or variables) hopefully detected early on
in the process of PCA(X). If outliers are the result of erroneous
sampling, or measuring, or if they represent truly aberrant data for
other reasons, they should be removed otherwise the model will not
be correct.
On the other hand, “outliers” may just as well be important, for
example if they are somewhat extreme, but still legitimate,
representatives for the data, in which case it is essential to keep them
in the model. Thus, if they represent a significant or important
phenomenon and they are removed, the model will also be
“incorrect”. It will lack, and will consequently be unable to explain,
this property which is in fact present in the data. The model will be an
equally poor approximation to the real world to which it is to be
applied sooner or later. This will of course appear to be a major
problem for new data analysts—that inferior model will be generated
if true outliers are to be included—and also if false outliers are
removed.
The good news is that outliers must either be one or the other.
The bad news is that it is always up to the analyst to decide whether
an outlier should be kept or discarded. In fact, the only problem is
that it will take some experience to get to know these things by their
appearances in the appropriate plots, but it is a feasible learning
assignment and it is one which is absolutely essential to master.
Textbook examples, exercises and personal experience will quickly
help to assume the necessary responsibility for detecting and
deleting outliers efficiently. It is essential to realise that there are
essentially only two major outlier detection modes:
1) Data analytical: the relative (geometrical) distribution of objects in,
e.g., the score plots is often all to go by. Decisions must be based
only on the data context and the data analysts analytical
experience.
2) Domain: i.e. problem-specific knowledge may in certain/many
other situations constitute a welcome reference with which to make
the difficult decision as to whether a particular object, or group of
objects, are outliers or not.
Score plots are particularly good for outlier detection. An outlying
object will appear in some score plot(s) as separate from the rest of
the objects, to a larger or smaller degree. Outliers are characterised
by excessively high scores (with either sign) as compared to all other
objects. Figure 6.11 shows two cases. In the left pane, the object with
an arrow is a potential outlier, but some observers may decide that it
nevertheless still “fits” with the general population, while the object in
the right pane is considerably more doubtful when assessed together
with all the remaining objects, and their internal trend. Observe that
such a relatively “clear” outlier will, in a major way, be significantly
responsible for the direction of a component (PC1 even, in the
illustrated case).
where X is the original data matrix, [X bar]is the mean of the data set
and W is the Covariance matrix of X.
The Hotelling’s T2 statistic is approximately F-distrubuted as
follows,
Figure 6.12: Gasoline octane number data score plot with Hotelling’s T2 ellipse at 25%
significance.
6.6.2 Leverage
Leverage is a measure of the effect of an object on a calculated
model and is a measure of the distance of an object from the model
centre. The leverage for an object is scaled between 0 and 1 with
extremely influential objects having a leverage close to 1, and a
“typical” (average) object having a leverage close to zero. Leverage is
calculated according to equation 6.2.
Figure 6.13: The influence plot and its use for detecting outliers and suspect objects.
where TNew is the projected score value for the new object XNew and P
is the loading vector(s) for the PCA model of Aopt.
In words, projection is the PCA equivalent of prediction using a
quantitative model (as discussed in chapter 7). It is a highly valuable
method for outlier detection post model development when the
model is used in real situations. Figure 6.14 shows the test set left out
of the alcohol mixture example projected onto the scores space of
the calibration set.
In Figure 6.14, green signifies the projected points and these
appear to lie in the same general space as the blue calibration
objects. The residual variance plot shows that the new sample
variance (green curve) follows the same pattern as the original model
(blue and red curves). It can be concluded from this evidence that the
projected objects are from the same population as the calibration
objects.
Figure 6.14: Alcohol mixture test set projected onto calibration set space model (left).
Figure 6.15: Outlier detection using validation scores for alcohol mixture analysis.
Figure 6.16: Stability plot of alcohol mixture analysis scores showing the influence object
A16 has on the model.
Figure 6.17: Loading plot of alcohol mixture analysis for PC2 showing the origin of the
outlying observation A16.
6.8.1 Purpose
To learn about outliers and how to recognise them from the score
plot and the influence plot, introduced earlier in this chapter.
Figure 6.18: P. Thy (left) trying out his hand in multivariate data analysis (for the very first
time) supervised by the first author of the present textbook—jointly elucidating the hidden
data structure of the Troodos data set.
One cardinal question is: are there more than one overall group of
samples?
If the rocks are all geochemically similar, one would expect that
the whole suite of rocks to have been formed geologically by the
same process, perhaps at the same time. If there are clear groups in
the locations of rocks, however, due to different geological
backgrounds, other conclusions about the formation of the area
might be drawn. This work was originally carried out to help settle a
major controversy regarding the entire geological history of the
Troodos massive, see Thy and Esbensen [5], who also show a further
pattern recognition data analysis aspect, not shown here; here we
focus on the initial PCA.
6.8.3 Analysis
Figure 6.18 provides the PCA overview of the Troodos where all
variables were auto scaled and the method of random cross
validation was used to gain an overview of the data.
The explained variance plot suggests that there are several
sources of variability in this data set, or that the possible common
petrogenetic process is not particularly simple. The score plot shows
that the general object population follows a distinct, narrowly defined
curved profile except for two distinctly different objects, namely
objects 65 and 66. Such extremely lonely samples might very well be
outliers.
The original problem context and the raw data must be
considered while working with these plots. It is also quite normal to
look at the original data matrix again when assessing potential
outliers. Whilst it would not be justified to regard either object 129 or
130 in a similar vein, it is a fairly safe initial bet that objects 65–66 are
indeed gross outliers, while object 129 is in all likelihood an extreme
end-member only. The status for object 130 needs further
investigation. Note that a too swift data analyst may possibly declare
all four objects 19, 20, 129 and 130 as extreme end-members.
The loading plot is shown as correlation loadings and it can be
seen that the variables most influential to PC2 (where objects 65 and
66 lie predominantly) are MnO, FeO and Al2O3. This also appears on
the straight loading plot in this case (not shown here).
The score plot in Figure 6.18 represents a typical picture of a
model where problems may occur. A “good model” spans the
variations in the data such that the resulting score plots show the
samples “well spread over the whole PC space”. In the present plot,
the samples are well spread in PC1, but only a very few samples
represent the major variations in PC2. All the other samples are
situated at, or very close to the origin in this direction (zero score
values in PC2), i.e. they have very little influence on the model in this
direction. It is also noted that the partitioning of the total variance
captured along PCs 1 and 2 is 54 and 23% respectively; thus, a fair
fraction of the total data variance is carried by each of the first two
components, but there may perhaps also be a need for higher-order
components.
The influence plot of Figure 6.18 shows the F-residuals vs
Hotelling’s T2. It is reproduced below in Figure 6.19, here illustrated
for PCs 1–4 in a four-pane view (even though the final PCA model will
be based on a smaller number of components).
Figure 6.21: PCA overview of Troodos data with objects 65 and 66 removed.
6.8.4 Summary
Figure 6.22: Summary illustration of scores and loadings. The direction of any PC is
described by its loading vector, which is composed of its directional coefficients. All objects
are projected down onto one, or more principal components, the coordinates of the resulting
footpoints are termed scores.
6.10 References
[1] Martens, H. and Næs, T. (1989). Multivariate Calibration. John
Wiley & Sons Ltd.
[2] Skoog, D.A., West, D.M. and Holler, F.J. (1988). Fundamentals
of Analytical Chemistry, 5th Edition. Saunders College
Publishing, pp. 466–472.
[3] Hotelling, H. (1933). “Analysis of a complex of statistical
variables into principal components”, J. Educ. Psychol. 24, 417–
441. https://doi.org/10.1037/h0071325
[4] Cook, R.D. and Weisberg, S. (1982). Residuals and Influence in
Regression. Chapman and Hall.
[5] Thy, P. and Esbensen, K.H. (1993), “Seafloor spreading and the
ophiolitic sequences of the Troodos complex: a principal
component analysis of lavas and dike compositions”, J.
Geophys. Res. 98(B7), 799–805.
https://doi.org/10.1029/93jb00695
7. Multivariate calibration
Figure 7.1: The process of multivariate calibration; establishing a regression model from
known X- and Y-data.
Figure 7.2: Using the multivariate regression model to predict new Y-values.
How well does a regression model fit to the X-data and to the Y-
data? How small are the modelling residuals? It is sometimes stated
that a good modelling fit also implies a good prediction ability, but
this is generally not necessarily so. It has to be proven to be so in
each case.
Initial modelling
Figure 7.4: Univariate regression: overlapping spectra in the wavelength regimen (x-axis).
7.7 Collinearity
Collinearity means that the X-variables are inter-correlated to a non-
negligible degree, i.e. that some of the X-variables (or in the worst
possible case, all of them) are linearly dependent to some degree; for
example,
x1 = f(x2,x3,…,xp)
Overall, the overlaid PC1 and PC2 loadings in Figure 7.11 show
that PC1 is the methanol/propanol direction and PC2 is the
methanol/propanol vs ethanol direction.
The last plot to review is the Predicted vs Reference plot which
shows the quality of the predictions with respect to the reference
measurements for each calibration and validation object. Predicted vs
Reference plots are discussed in more detail in section 7.10.10. For
this example, the linearity of the Predicted vs Reference plot is
visually very strong and the calibration and validation points lie in
their expected spaces.
Regression coefficients and Predicted vs Reference plots for the
responses ethanol and propanol are provided in Figure 7.12 to
complete the analysis.
By comparison of all of the regression coefficients, it is apparent
that each alcohol component is modelled by different specific and
individual parts of the spectrum. This is an outcome of using a
designed experiment to generate the calibration set which is well
suited to support multivariate regression models for several Y-
variables.
Figure 7.12: Regression coefficients and predicted vs reference plots for responses ethanol
and propanol, alcohol mixture analysis.
Thus, what might at first sight have been viewed as two parallel
sets of independent PCA decompositions, is in fact based on these
interchanged score vectors instead. In this way, the goal of modelling
the X- and Y-space inter-dependently has been achieved. By
balancing both the X- and Y-information, PLSR actively reduces the
influence of large X-variations which do not correlate with Y, and so
does the job of removing the problem of the two-stage PCR
weakness outlined above.
In many algorithmic implementations of PLSR, a slope of 1 in the
relationship between t and u is assumed, thus Y can alternatively be
represented as Y = TQT + F. Some chemometricians prefer the
simpler relationship based on a common object designation; T, in the
model equations, i.e. working with the Y-space without u-score
vectors as it were. While this will result in numerically identical
predictions, a.o. the possibility of insight into the operative X–Y part
models is completely lost with this option, however, for which
reasons most chemometricians prefer the full t–u approach. The
authors of this textbook are very strongly advocating this latter
approach precisely because of its vastly superior model
interpretability features. “There should be no PLSR modelling without
full model understanding”.
The Non-Iterative Partial Alternating Least Squares (NIPALS)
algorithm was historically developed first [6]. It will be outlined in full
algebraic detail below with reference to the standard PCA algorithm
which was introduced in the same detail in chapter 4, section 4.14.
2) (w is normalised)
3) tf = Xfwf
4) (q is normalised)
5) uf = Yfqf
6) Convergence: if |tf,new – tf,old| < convergence limit, stop; else go to
step 2. (uf may be used alternatively)
7)
8) (PLSR inner relation)
9)
10) f = f + 1
Repeat 1 through 10 until f = A (optimum number of PLSR factors
by validation).
All PLSR calibrations result in two sets of X-loadings for the same
model. They are called loadings, P, and loading weights (or just
weights or PLSR-weights), W.
The P-loadings are very much like the well-known PCA-loadings;
they express the relationships between the raw data matrix X and its
scores, T. These may be used and interpret in the same way as in
PCA or PCR, so long as it is remembered that the scores have in fact
been calculated by PLSR (and are not necessarily orthogonal).
In many PLSR applications P and W are quite similar. When this is
the case, it is because the dominant data structures in X “happen” to
be directed more or less along the same directions as those with
maximum correlation to Y. In all such cases the difference is not very
interesting—the p and w vectors are pretty much identical. The
duality between P and W will only be important in the situation where
the P and the W directions differ significantly, however. This
difference between loadings and loading weights will be
demonstrated in the octane in gasoline example presented later in
this chapter.
The loading weights, W, represent the effective loadings directly
connected to building the desired regression relationship between X
and Y. Vector w1 characterises the first PLSR-factor direction in X-
space, which is the direction onto which all the objects are projected
in a PLSR model. Principally, this direction is not identical to the p1
direction, but differences between p1 and w1 may vary. It is precisely
this difference that provides information on how much the Y-
guidance has influenced the decomposition of X; one may think of
the PCA t-score as being tilted because of this PLSR-guidance. One
illuminating way to display this relation is to plot both alternative
loadings in the same 1-D loading plot or as side-by-side plots.
The sequential updating in the PLSR-algorithm implies that a
similar relationship to that between p1 and w1 also holds for the
higher-order PLSR-factors 2, 3,... The w-vectors that make up the W
matrix are all mutually orthogonal. Therefore, inspection and
interpretation of the loading weights in 2-D vector plots, e.g. w1
versus w2, can be performed exactly as was done for PCA (using p–p
plots), while remembering that W relates to the regression between X
and Y. Thus, W tells the effective PLSR “X-space loading story” with
respect to the inter-variable relationships.
In PLS2 models, there is also a set of Y-loadings, Q, which are the
regression coefficients from the Y-variables onto the individual
scores, U, Q and W. These may be used to interpret relationships
between the X- and Y-variables, and to interpret the patterns in the
score plots related to these loadings. The specific use of these
double sets of scores (T, U) and loadings (W, Q) are illustrated by the
practical PLSR-analytical examples to be presented below.
The fact that both P and W are important, however, is clear from
construction of the formal regression equation Y = XB from any PLSR
solution with A factors. This B “compaction” regression matrix is
derived as follows:
B = W(PTW)–1QT
This B-matrix is often used for practical (numerical) prediction
purposes, if there is no specific need for interpretation of the model in
question.
Now for the NIPALS algorithm for the PLS1 approach, in which the Y-
matrix is in the form of a single y-vector (Q = 1). Because the PLS2
algorithm was discussed in detail above, it will be very easy both to
understand PLS1’s specific features, but especially so to appreciate
its particularly simple “reductionist” position in comparison to PLS2.
In fact, by noting the matrix–vector substitution, Y → y, it will be
possible to follow how the relevant steps in the corresponding PLS2
algorithm that concerns the score vector u, all simply collapse, which
the, at first, rather surprising result that the convergence feature is
actually made redundant. The result is a much simpler, non-iterative
calculation procedure. As usual, centring and scaling first:
1) Centre and scale both the X- and Y-matrices appropriately.
2) Index initialisation, f: f = 1; Xf = X; yf = y
3) The y-vector is now its own proxy “u-vector” (there is only one Y-
column)
a) (w is normalised)
b) tf = Xfwf
c)
d)
e) Xf + 1 = Xf – tfpTf yf + 1 = yf – qftTf
f) f = f + 1
Repeat until f = Aopt (optimum number of PLSR-factors by
validation).
The PLS1 algorithm, and procedure, is as simple as this. There are
no other bells or whistles. Because of its computational simplicity
PLS1 is very easy to perform, but there are other, more important—
indeed crucial—reasons why PLS1 has become the singular most
important multivariate regression method, which will be laid out
below.
7.10.1 Purpose
The production of petroleum (gasoline) in major oil refineries is big
business, but also one in which even small inefficiencies can translate
to large operational profit losses. One of the major properties in
gasoline production is to optimise the blend of products within tight
octane number specifications. Octane number is a measure of the
burning efficiency of the fuel, with higher octane number gasolines
forming the basis of premium fuels (where the highest profit margin
can be made) and lower octane number fuels going into more
standard products.
It is not only the final octane number that requires optimisation,
but also the individual component amounts of the blend feedstocks
used to attain optimal octane numbers, i.e. blend components that
typically display higher octane numbers are either expensive (made
by synthesis), including components such as alkylates, or are
environmentally dangerous and must be regulated (i.e. benzene and
other aromatic compounds). In modern refineries, where thousands of
litres of gasoline are produced every minute, if a deviation from recipe
is not detected in a timely manner, then gasoline of higher
performance may be blended into a lower grade product (this would
be to the consumer’s benefit) whereas, in the opposite case, lower
quality gasoline is being produced for premium product (the
consumer’s disadvantage).
It has been estimated that every 0.3 octane number deviation
resulting from inaccurate or inefficient blending costs a refinery
around 0.20 USD per barrel of product (2016 figures). Considering
that a small refinery produces 100,000 barrels (bbl) per day and a
large refinery up to 1.2 mil bbl per day, such “give away” costs range
between 20K and 240K USD per day of operation. If left in such an
un-optimised state even at 0.20 USD giveaway per barrel, over a
continuous year, a single refinery may well give away a very
significant part of its profits. This is why methods such as on-line
spectroscopy are highly desirable particularly for implementation into
blending operations. These systems produce real or near-real time
data that can be used to control feedstock flows into the blend
headers and control the gasoline production to tight specifications.
The following example shows how a model for octane number
prediction can be developed using NIR spectroscopy and PLSR.
7.10.3 Tasks
This data set will be analysed using The Unscrambler® PLSR
capabilities and a description of the steps involved will be outlined in
detail in this example. The main tasks are to develop a calibration
model for octane number using the training set and validate the
model using the test set.
Figure 7.16: PCA overview of the NIR octane in gasoline data set.
It was observed in the PCA model that the spectral features that
separate the data into two groups occur in the region 1350–1550 nm
and from the initial PLSR, the spectral features related to octane are
in the region 1100–1350 nm. Figure 7.19 provides the loadings and
loading weights for factors 1 and 2 of the PLSR model displayed in a
four-pane overview.
Figure 7.19: Assessment of PLSR factor loadings and loading weights for octane number in
gasoline data set.
Figure 7.20: a) Calibration and b) validation predicted vs reference plots for octane in
gasoline data.
7.10.13 Residuals in X
7.10.14 Q-residuals
Q-residuals were first introduced by Jackson and Mudholkar [10] to
detect outliers that can be the result of,
1) Too few components used to adequately describe the original data
X.
2) The samples are truly outliers for a specific model.
Q-residuals are calculated from the regular X-residual as a
squared sum,
where Qa is the critical value for the Q-distribution and the remainder
of the terms can be found in the standard text by Jackson [11].
7.10.15 F-residuals
PRESS
An alternative error measure is the Predicted Residual Sums of
Squares, PRESS (equation 7.17):
Bias
The bias is the average value of the difference between the reference
and predicted values for a carefully specified set of replicated sample
measurements (equation 7.16).
Figure 7.25: PLSR overview of octane in gasoline model with suspect samples removed.
Elements 26 13 24 13 26 13
Number of
3 2 3
PLSR factors
Figure 7.29: X–Y relationship outlier plot for the variable elimination octane in gasoline
model.
X = TPT + E
and
Y = TbT + F
X = TPT + E
and
Y = TQT + F
where TA,New are the projected scores for the optimal number of
components/factors, XNew is the new data being presented to the
model and PA,Model are the loadings of the model for the optimal
number of components/factors. By assessing the projected scores,
the new X-data can be tested to see if it is a reasonable assumption
that they originate from the same population as the calibration model.
If the projected scores lie outside of the model space, there is reason
to treat the Y-predicted value as suspect, until/unless proven
otherwise. For PLSR models, it is the Aopt rows of the W-matrix that
are to be used.
7.12.3 Y-deviation
Figure 7.30: Predicted with Y-deviation plot for the variable selection model applied to the
test set, octane number in gasoline data.
Figure 7.31: Predicted with Y-deviation plot for the sample elimination model applied to the
test set, octane number in gasoline data.
The inlier vs outlier plot can be interpreted in the same way as the
influence plot (refer to chapter 6, section 6.6.4. Predicted objects that
lie within the confidence bounds of the Inlier and Hotelling’s T2
statistics are considered to be similar to the calibration objects.
Those that exceed the Hotelling’s T2 limit only are considered to be
high leverage objects. Objects that exceed the inlier statistic bound
only are samples that could be added to the model to make the
model more robust. Objects that simultaneously lie outside of the
inlier and Hotelling’s T2 bounds should be treated as outliers and
further investigated or deleted outright.
The inlier vs outlier plot for the sample elimination training data set
model is provided in Figure 7.33.
When the sample elimination model is applied to the test set, it is
better able to distinguish the samples with additive as outliers. The
inlier vs outlier plot also shows that one object may be a potential
inlier.
This example highlights that the model development process does
not stop when the initial model has been developed and
implemented. All multivariate models used for real world applications
must have a lifecycle maintenance scheme built around them that
defines when,
Figure 7.32: Inlier vs outlier plot for octane in gasoline variable selection model applied to
test set.
Figure 7.33: Inlier vs outlier plot for octane in gasoline sample elimination model applied to
test set.
T(m) = TmCm
[PT,Q](m) = Cm–1[PmT,QmT]
Context
This data set was introduced in chapter 5 on preprocessing and was
taken from the work performed by Martens, Nielsen and Engelsen
[19]. It is well known that scatter effects dominate spectral data
collection of solid samples. In the case of transmission spectroscopy,
not only is scattering a major effect, but also pathlength effects and
packing density of solid samples influences the shape and slope of
the spectra. The objective of the current investigation is to show that
no matter what sample thickness and presentation, the same
constituent composition should be able to be modelled from the
samples.
Figure 7.34: Score plot for Factor-1 vs Factor-2, sample 10 is marked (lower right).
Data set
Data analysis
The first step, as always is to visualise the raw data. The line plot in
Figure 7.37 shows that this data set is highly diverse chemically, i.e.
the spectra do not show a similar profile, but are varying in shape. It
is also observed from Figure 7.37 that there is a large effect of
packing density as evidenced by the large baseline offsets between
the spectra. It is known that in the short-range wavelength region of
the NIR, diffuse reflectance is not a dominant effect and specular
reflectance is the mechanism of light scattering, however, this is not
the case for transmission data and therefore scattering effects are as
relevant to transmission as they are for diffuse reflectance in the
longer wavelength NIR region.
To determine which preprocessing method to use, it was shown in
chapter 5 that using the scatter effects plot, this data set did not
show constant straight line properties when plotted against the mean
spectrum (due to the diverse chemical nature of the samples). The
EMSC transform was better able to reduce the scatter and packing
effects in the data and the application of the mEMSC preprocessing
resulted in a very similarly looking data set.
PLSR models for gluten content using the raw, MSC, EMSC and
mEMSC were developed and the results obtained are described in
the following sections.
In all cases, when a model can be applied to raw data, this is the best
option, provided a reasonable number of components/factors are
used in the model. For a binary mixture, the optimal number of PLSR
factors would be 1, however, 2 factors (possibly 3) may also be
acceptable. Any more than this number would likely indicate that the
model is trying to base predictions around random or chance
correlation structures in the data set.
Figure 7.37: Raw NIR transmission spectra of gluten–starch mixtures.
The original objective for modelling this data set was to show that
through the effective use of meaningful preprocessing, a simple one-
factor PLSR model should result from the analysis of a binary mixture
of powders with diverse chemical information contained in their
spectra. Table 7.4 summarises the statistical parameters for each
model and also provides the statistics of each model for a one-factor
PLSR situation only for comparative purposes.
It is not the present authors’ intent that the reader follows prescriptive
rules for model development rigidly (although these summarise a
wealth of empirical experience). However, the use and adaptation of
the following list in the assessment of model quality will help to
achieve the most robust modelling situations for “typical data sets”.
The following list details reasons for caution:
Optimised # Factors 3 6 3 1
1-Factor # Factors 1 1 1 1
The prediction error is “too high” for the problem at hand (w.r.t
external knowledge).
The residual variance displays an increase before the minimum, or
does not decrease at all (in PLSR).
There are conspicuous “isolated objects” (possible outliers), in
scores, residuals or influence plots, or groups of similar outlying
objects.
Similarity of isolated, outlying variables, although the interpretation
of unnecessary, or bad, variables is not identical to that pertaining
to objects.
The distribution of objects in T vs U score-plots (X–Y relation outlier
plot) has a non-linear shape or shows a worrisome presence of
groups (“clumpiness”) a.o. There are systematic patterns in the
residuals.
New data are not predicted well.
Outliers are (still) present in the training data (and/or in the test set
data).
Data are not representative: for instance, the calibration data are
not representative of future prediction conditions, or the validation
data are not representative with respect to the calibration data.
There are several other possible reasons for this characterisation as
well.¶
Unsatisfactory validation (wrong validation method).
Lack of variability in calibration data set range(s); ditto re test set
range.
Need for pretreatment of raw data (e.g. linearisation).
Inhomogeneous data (subgroups, “clumpy data”).
Errors in sample preparation.
Systematic errors in experiments (significant bias, either sampling
bias and/or analytical).
Instrument errors, e.g. drift, upset.
Lack of information—there is in fact no X–Y relationship (always a
possibility).
Strongly non-linear X–Y relationships.
7.16 References
[1] Draper, N.R. and Smith, H. (1998). Applied Regression Analysis,
3rd Edn. John Wiley & Sons.
https://doi.org/10.1002/9781118625590
[2] Vandeginste, B.G.M., Massart, D.L., Buydens, L.M.C., De Jong,
S., Lewi, P.J. and Smeyers-Verbeke, J. (1998). “Quantitative
structure activity relationships (QSAR)”, in Handbook of
Chemometrics and Qualimetrics, Part B. Elsevier, Ch. 37, pp.
383–420. https://doi.org/10.1016/S0922-3487(98)80047-6
[3] Naes, T. and Isaksson, T. (1992). “Locally weighted regression
in diffuse near-infrared transmittance spectroscopy”, Appl.
Spectrosc. 46, 34–43.
https://doi.org/10.1366/0003702924444344
[4] Höskuldsson, A. (1996). Prediction methods in Science and
Technology, Vol. 1. Basic Theory. Thor Publishing, Denmark.
[5] Swarbrick, B. (2016). “Chemometrics for near infrared
spectroscopy”, NIR news 27(1), 39–40.
https://doi.org/10.1255/nirn.1584
[6] Wold, H. (1966). “Estimation of principal components and
related models by iterative least squares”, in Multivariate
Analysis, Ed by Krishnaiah, P.R. Academic Press, NY.
[7] Ergon, R. (1992). “PLS score-loading correspondence and a B-
orthogonal factorisation”, J. Chemometr. 16, 368–373.
https://doi.org/10.1002/cem.736
[8] Ergon, R., Halstenen, M. and Esbensen, K.H. (2011). “Model
choice and squared prediction errors in PLS regression”, J.
Chemometr. 25, 301–312. https://doi.org/10.1002/cem.1356
[9] Cook, R.D. and Weisberg, S. (1982). Residuals and Influence in
Regression. Chapman and Hall.
[10] Jackson, J.E. and Mudholkar, G.S. (1979). “Control procedures
for residuals associated with principal component analysis”,
Technometrics 21, 341–349.
https://doi.org/10.1080/00401706.1979.10489779
[11] Jackson, J.E. (1991). A User’s Guide to Principal Components.
Wiley series in probability and mathematical statistics. Applied
probability and statistics. Wiley.
https://doi.org/10.1002/0471725331
[12] Swarbrick, B. and Westad, F. (2016). “An overview of
chemometrics for the engineering and measurement sciences”,
in Handbook of Measurement in Science and Engineering, Ed
by Kutz, J. John Wiley & Sons, Hoboken, NJ.
https://doi.org/10.1002/9781119244752.ch65
[13] Esbensen, K.H. and Geladi, P. (2010). “Principles of proper
validation: use and abuse of re-sampling for validation”, J.
Chemometr. 24, 168–187. https://doi.org/10.1002/cem.1310
[14] ICH Harmonized Tripartate Guideline Q2(R1) (1997). “Validation
of analytical procedures: text and methodology”, Federal
Register 62(96), 27463–27467.
[15] Broad, N., Graham, P., Hailey, P., Hardy, A., Holland, S.,
Hughes, S., Lee, D., Prebble, K., Salton, N. and Warren, P.
(2002). “Guidelines for the development and validation of near-
infrared spectroscopic methods in the pharmaceutical industry”,
in Handbook of Vibrational Spectroscopy, Ed by Chalmers, J.M.
and Griffiths, P.R. John Wiley & Sons.
https://doi.org/10.1002/0470027320.s8303
[16] Westad, F., Byström, M. and Martens, H. (1999). “Modified jack-
knifing in multivariate regression for variable selection and
model stability”, in Near Infrared Spectroscopy: Proceedings of
the 9th International Conference, Ed by Davies, A.M.C. and
Giangiacomo, R. NIR Publications, Chichester.
[17] Esbensen, K.H. and Mortensen, P. (2010). “Process sampling
(Theory of Sampling, TOS)—the missing link in process
analytical technology (PAT)”, in Process Analytical Technology,
2nd Edn, Ed by Bakeev, K.A. Wiley, pp. 37–80.
https://doi.org/10.1002/9780470689592.ch3
[18] Efron, B. (1982). “The Jacknife, the Bootstrap and Other
Resampling Plans”, CBMS-NSF Regional Conference Series in
Applied Mathematics, Volume 38. Capital City Press, VT, USA.
[19] Martens, H., Nielsen, J.P. and Engelsen, S.B. (2003). “Light
scattering and light absorbance separated by extended
multiplicative signal correction. Application to near-infrared
transmission analysis of powder mixtures”, Anal. Chem. 75,
394–404. https://doi.org/10.1021/ac020194w
[20] Romanach, R. and Esbensen, K.H. (2016). “Theory of Sampling
(TOS) for development of spectroscopic calibration models”,
Amer. Pharm. Rev. September/November,
Spectroscopy/Review section, pp. 1–3.
* “The same sample” is the view from the analytical laboratory and the computer on which
the chemometric software resides. It is imperative not to forget, however, that “samples”
always originate through some process in which sampling is involved. Sometimes this full
understanding has consequences for the ease with which a relevant set of “the same
samples” can be, or will have to be produced. This reminder points to the broader context in
which all multivariate calibrations exist.
† See footnote * regarding the fact that in real life, analytical results most specifically do not
exist only as “data” i.e. as a potential collection of all possible analytical data only, which can
be sampled easy enough in the strict statistical sense, while they were in fact meticulously
arrived at by the complete pathway from lot-to-analysis, a framework which has the utmost
emphasis in this book.
‡There is a burdensome tradition in statistics and data analysis of not always using a strict
terminology. A “factor” is strictly speaking a factor in Factor Analysis, whereas a
“component” is a component in PCA. Both are linear combinations of all the P variables
making up the X matrix, or ditto re the Q variables in the Y matrix. Unfortunately, in the
international literature factors and components are rather often used inter alia quite liberally.
Thus “factor” is sometimes used liberally when meaning PCA, PCR, PLS components. Blame
the historical development of chemometrics…
§The systematics of vector and matrix names in the algorithmic overviews of methods were
laid down from the very first beginnings of chemometrics in a very helpful way: “X→Y, look
for the next letter in the alphabet”.
¶ Calling data “representative”, or “non-representative” is clearly one of the most complex
justification tasks one can meet. Representativity is a term that can be (should be) applied to
the sampling process, to the sample preprocessing steps in the laboratory, to the analytical
process and, finally, to the choice of model (model selection). All this takes place before the
“data” originated; thus, stipulating a matrix as indeed containing “representative data” is a
statement for which one can get into the hot seat very quickly. One better know whereof one
is speaking—in all aspects. Very often one finds an extremely frivolous use of this crucial
term in the pertinent literature, when in fact used mostly as wishful thinking….
8. Principles of Proper Validation
(PPV)
8.1 Introduction
A set of generic Principles of Proper Validation (PPV) is presented,
based on five distinctions:
i) The PPV are universal and apply to all situations in which validation
assessment of performance is desired: modelling, prediction,
classification, time series forecasting.
ii) The key element behind PPV is the Theory of Sampling (TOS), which
provides insight into all variance-generating factors, especially the
“Incorrect Sampling Errors” (ISE). If not properly eliminated, these
are responsible for an inconstant sampling bias, for which no
correction is possible, contrary to the widespread statistical
tradition for “bias correction” (see also chapter 3). A sampling bias
wreaks havoc with all types of validation—except test set
validation. The sampling strategy should always address how
known (and unknown) sources of variation can meaningfully be
included in the first dataset which is to be modelled, the training
set. Thus, sampling includes qualitative information about time,
location, batch ID and other qualitative information related to the
sampling strategy, and not only the physical procedure. Such a
comprehensive overview is imperative for a reliable estimation of
future prediction or classification performance.
iii) Validation cannot be understood solely by focusing on the
method(s) of validation—it is not enough to be acquainted only with
a particular validation scheme, algorithm or implementation as has
been a longstanding tradition in chemometrics. Validation must be
based on full knowledge of the underlying definitions, objectives,
methods, effects and consequences of the full sampling–reference
analysis–data analysis process.
iv) Analysis of the most general validation objectives leads to the
conclusion that there is one valid paradigm only: test set validation.
In this chapter, the most important alternative validation
approaches are discussed, critiqued and rejected in this
perspective.
v) Contrary to contemporary chemometric practices and validation
myths, cross-validation is unjustified in the form of a one-for-all
procedure for all data sets. Within its own methodological scope,
cross-validation is shown to be but a suboptimal simulation of test
set validation, as it is based on one data set only (the training data
set). However, such a “first” singular data set could, occasionally,
be representative of future variation, but one would never know
whether this is the case within the one-sample-set context alone;
only external experience or evidence would be decisive. Many re-
sampling validation methods suffer from this principal deficiency.
Thus, while there are cases in which cross-validation finds excellent
use, for example, categorical segment cross-validation (in which
categories can be batches, seasons, alternative models or pre-
treatments), see below, such cases represent only special
circumstances and no generalisation regarding validation principles
can be made hereupon.
This chapter shows how a second data set (test set, external
validation set) constitutes a minimum critical success factor for
inclusion of the sampling errors incurred in all “future” situations in
which the validated model is to perform. From this follows that all re-
sampling validation approaches based on a singular data set only (for
example a data set sampled on one day only, or with one instrument
only, or addressing one batch of raw materials only) should logically
be terminated, or used only with full scientific understanding and
disclosure of their detrimental limitations and consequences. This
chapter builds on a comprehensive analysis of validation in Esbensen
and Geladi [1]; see also Westad and Marini [2].
In the central case of PLSR, a call is here made for stringent
commitment to test-set validation based on graphical inspection of
pertinent t–u plots for optimal understanding of the underlying X–Y
data structures and interrelationships for validation guidance. The t–u
visualisation, or similar regarding other multivariate data modelling
methods in need of validation, is critical in order to stop continuation
of decades of “blind” use of a one-for-all procedure cross-validation,
a state of affairs that has led to quite some confusion among
generations of chemometricians.
Figure 8.4: Illustration of a training set and test sets of progressively less-and-less overlap.
8.15 References
[1] Esbensen, K.H. and Geladi, P. (2010). “Principles of proper
validation: use and abuse of re-sampling for validation”, J.
Chemometr. 24, 168–187. https://doi.org/10.1002/cem.1310
[2] Westad, F. and Marini, F. (2015). “Validation of chemometric
models—a tutorial”, Anal. Chem. Acta. 893, 14–24.
https://doi.org/10.1016/j.aca.2015.06.056
[3] Martens, M. and Martens, H. (2001). Multivariate Analysis of
Quality. An Introduction. Wiley, Chichester, p. 445.
[4] Høskuldsson, A. (1996). Prediction Methods in the Sciences.
Thor Publishing. Copenhagen.
[5] Press, W.H., Teukolsky, S.A., Vetterling, W.T. and Flannery, B.P.
(2007). Numerical Recipes, The Art of Scientific Computing, 3rd
Edn. Cambridge Press, NY, p. 777.
[6] Wonacott, T., and Wonacott, R. (1990). Introductory Statistics,
5th Edn. Wiley, New York.
[7] Devore, J. (1995). Probability and Statistics for Engineering and
the Sciences, 4th Edn. Duxbury Press, Belmont, CA.
[8] Martens, H. and Naes, T. (1989). Multivariate Calibration. Wiley,
Chichester.
[9] Esbensen, K.H. and Paasch-Mortensen, P. (2010). “Process
sampling (Theory of Sampling)—the missing link in process
analytical technologies (PAT)”, Process Analytical Technologies,
2nd Edn, Ed by Bakeev, K. Wiley-Blackwell, Oxford.
https://doi.org/10.1002/9780470689592.ch3
[10] US FDA, Guidance for Industry—Process Validation: General
Principles and Practices.
http://www.fda.gov/downloads/Drugs/Guidances/UCM070336.pdf
[Accessed 6 January 2017].
[11] Esbensen, K.H. and Lied, T.T. (2007). “Principles of image
cross-validation (ICV): representative segmentation of image
data structures”, in Techniques and Applications of
Hyperspectral Image Analysis, Ed by Grahn, H.F. and Geladi, P.
Wiley, Chichester, Ch. 7, pp. 155–180.
https://doi.org/10.1002/9780470010884.ch7
[12] Martens, H. and Dardenne, P. (1998). “Validation and
verification of regression in small data sets”, Chemometr. Intell.
Lab. Syst. 44, 99–121. https://doi.org/10.1016/S0169-
7439(98)00167-1
* In chemometrics, there has been a key terminology confusion regarding the terms, sample,
object, observation, measurement a.o. The duality regarding “sample” in the statistical vs the
TOS contexts is covered comprehensively in this book. An “observation” can be understood
as a passive measurement. A physical sample may result in several “replicate”
measurements, which then would become individual objects when represented in a data
matrix. Chapters 3, 8 and 9 have made a profound effort to clear up these confused
terminology issues.
† Chemometrics owe Høskuldsson a very great depth of gratitude for his seminal 1996
treatise; the H-principle name may very well connote the last name of this chemometric
author as well.
‡ One test set is the minimum requirement… but more can, of course, also be contemplated
in specific situations, especially in cases where data set structures are varying more than
what is comfortable. This issue is particularly relevant for PAT implementations and other
process data modelling realms, see chapter 13.
§ Note that in reality the model is only “robust” with respect to alternative validation methods
or strategies.
¶ In a recent meeting of the Australian Near Infrared Spectroscopy Group (ANISG), the
respected chemometrician Professor Tom Fearn presented a paper entitled: “Crass-
Validation”—a term that well aligns with the current argumentation in this textbook.
** However, there is a paradox when one set of objects is collected on one day, with one raw
material etc., there is still no information about the samples as basis for systematic validation
in the sense of “conceptual cross-validation”. In this case only random cross-validation or
setting aside one part of the objects as a test-set are real alternatives. A random split into a
calibration and a test set will not reveal if the model is stable towards future sources of
variation. Again, only a well thought out training data set will allow validation to address all
the relevant issues; the specific cross-validation method alone will be insufficient.
9. Replication—replicates—but of
what?
9.1 Introduction
From the discipline of experimental design (design of experiments,
DOE, chapter 11), comes a well-organised strict understanding and
terminology for “replicate measurement”, because of the rigidly
controlled situation surrounding the actual design. For example, in
the situation of chemical synthesis influenced by several experimental
factors (say, temperature, pressure, concentration of co-factors), it is
easy to understand what a replicate measurement means: repeat the
synthesis experiment under identical conditions for these controllable
factors and replicate the outcome measurement, for example, the
yield. By definition of a designed experiment, care has been taken to
randomise all other potential factors, in which case the variance of
the experimental results, be it small or large, is supposed to furnish a
measure of the “Total Analytical Uncertainty”. Upon reflection,
however, this variance also tells of the uncertainty contributions
stemming from other influences, for example, from small-scale
sampling of reactants involved, which may not necessarily represent
“homogeneous stocks”, but more likely are of uniform composition
only, see chapter 3. Added uncertainty contributions may also arise
from resetting the experimental setup, i.e. to what precision can one
“reset” parameters such as temperature, pressure or concentration
levels of co-factor chemical species after having turned the setup off
and cleaned all the experimental equipment (perhaps even waiting
until next week) before “replicating”? However, such uncertainty
contributions are usually considered insignificant because of the
“controllable” situation attending DOE.
There are, however, many other scenarios preceding data analysis
that far from parallel this nicely bracketed situation of a controlled
experiment. Indeed, most data sets do not originate from within the
complacent four walls of an analytical laboratory only, but from
sampling of heterogeneous systems and processes from all of
science, technology and industry. What are described below
constitute the opposing end of a full spectrum of possibilities in which
the researcher/data analysts must recognise as significant sampling,
handling and preparation errors in addition to Total Analytical Error
(TAE). The issue can most conveniently be organised around one key
concept: What is meant by “replicate samples”? Note that one
physical sample may end up being represented in a data matrix as
several objects etc. with a very real danger of potential confusion.* To
add to the complexities, replication may also be carried out at other
levels, for example, related to sampling of multiple batches, lots or
production units, sampling from different seasons or from different
instruments, perhaps carried out by different operators. All these
higher-order replication issues are discussed in more detail in chapter
8. For each of these scenarios, it is imperative that the reader is
offered full disclosure of the stage, level and intensity of replication.
Upon reflection, this issue will appear more complex than what
may seem the case at first sight, indeed it merits careful definition,
contemplation and a strict terminology among other reasons because
it is also intimately related to the validation issue presented in chapter
8. There has been much confusion (due to unawareness and
sometimes neglect) because of far too vague or incomplete
definitions of what it is exactly that is replicated.
As the case in point, what is meant by “replicated samples”?
With reference to TOS, it will be appreciated that “replication” can
concern (at least) the following alternative scenarios:
Figure 9.5: Hierarchical replication setup. A separate RE(r) for each sampling stage allows full
decomposition of RE(r) variance into stage components.
Figure 9.6: Analytical vs sampling bias. The analytical bias is per definition always assumed
to be constant and can therefore be subjected to a statistical bias-correction (lower-right
panel). The sampling bias is of a fundamentally different nature, however, due to
heterogeneity. The sampling bias varies in magnitude every time it is estimated as a
consequence of the heterogeneous nature of the lot/system and can therefore never be
corrected for (upper-right panel). If estimated one more time the sampling variability would
again be both biased and imprecise (but would constitute a different point swarm location
and disposition in the dart board illustration metaphor used here). Instead the sampling bias
has to be eliminated, the tools for which is to be found in TOS.
9.7 References
[1] Pitard, F. (2009). Pierre Gy’s Theory of Sampling and C.O.
Ingamell’s Poisson Process Approach. Pathways to
Representative Sampling and Appropriate Industrial Standards.
Doctoral thesis, Aalborg University. ISBN 978-87-7606-032-9
(available from the author at [email protected])
[2] DS 3077. Representative Sampling—Horizontal Standard (2013).
Danish Standards, www.ds.dk
[3] Esbensen, K.H, Geladi, P. and Larsen, A. (2013). “Mythbusters
in Chemometrics: The replication Myth 1”, NIR news 24(1), 17–
20. https://doi.org/10.1255/nirn.1390
[4] Esbensen, K.H., Geladi, P. and Larsen, A. (2013), “Mythbusters
in Chemometrics, 6: The Replication Myth 2: Quantifying
empirical sampling plus analysis variability”, NIR news 24(3),
15–19. https://doi.org/10.1255/nirn.1364
[5] Esbensen, K.H. and Romañach, R.J. (2015). “Proper sampling,
total measurement uncertainty, variographic analysis & fit-for-
purpose acceptance levels for pharmaceutical mixing
monitoring”, TOS forum Issue 5, 25–30.
https://doi.org/10.1255/tosf.68
[6] “ICH Harmonized Tripartite Guideline Q2(R1), Validation of
Analytical Procedures: Text and Methodology” (1997). Federal
Register 62(96), 27463–7.
[7] Esbensen, K.H. and Wagner, C. (2014). “Theory of Sampling
(TOS) versus Measurement Uncertainty (MU) – a call for
integration”, Trends Anal. Chem. 57, 93–106.
https://doi.org/10.1016/j.trac.2014.02.007
[8] Esbensen, K.H. and Geladi, P. (2010). “Principles of Proper
Validation: use and abuse of re-sampling for validation”, J.
Chemometr. 24, 168–187. https://doi.org/10.1002/cem.1310
[9] Esbensen, K.H., Geladi, P. and Larsen, A. (2014). “The RPD
myth…”, NIR news 25(5), 24–28.
https://doi.org/10.1255/nirn.1462
* This chapter is intended to deal comprehensively with the confusion surrounding all issues
of “replication”. A general argument will be presented covering the most common and also
less common scenarios.
† Taguchi approach: http://en.wikipedia.org/wiki/Taguchi_methods
‡ “There is always an uncertainty, regardless of how small it is, between the true, unknown
content aL of the lot L and the true, unknown content of the sample, aS. … tradition has
established the word ‘error’ as common practice, though it implies a mistake that could have
been prevented, while statisticians prefer the word ‘uncertainty’ which implies no
responsibility. However, in practice, as demonstrated in the Theory of Sampling, there are
both sampling errors, and sampling uncertainties. Sampling errors can easily be minimised,
while sampling uncertainty for a pre-selected sampling protocol in inevitable. …. Because
the word ‘uncertainty’ is not strong enough, the word ‘error’ has been selected as current
usage in the Theory of Sampling, making it very clear it does not necessarily imply a sense of
culpability”; quoted from Pitard [1] p. 33 who graciously informs that this statement rightly
originates with Pierre Gy (in a monograph written in French, 1967).
§ There has been an extensive debate at meetings and in the literature within the
international sampling community as to setting up a (completely) general RSV threshold.
Opinions have slowly converged to accept a suggestion, which originates from the
characteristics of the Poisson distribution (sampling can in a certain sense be likened to a
Poisson selection process), that a RSV larger than 20% signifies that the average of repeated
sampling-and-analysis is out of control. Be aware, however, that this is an attempt to
characterise all the world’s extremely different materials, lots and processes with one
singular threshold—a very dangerous simplification! More on this important issue can be
found in DS 3077 [2] and in Pitard [1].
¶ There exist many types of materials with significantly different heterogeneity levels. The
general threshold for RSV (20–30%) refers to significantly heterogeneous materials, a very
wide and diverse class of materials, for example, mineralisation, ores, pollutant effects,
industrial aggregates (building materials, waste, biomass …), the list is exhaustive. There are
also many other types of materials which exhibit less heterogeneity, for which a lower
threshold will be relevant, say at the 10%, or 5% level, or even lower. It is emphasised that
the demand for a general RSV is a demand which cannot be met universally. It is necessary
to invoke common sense when making RSV operational in a distinct problem-dependent
fashion [2–4].
** A classical case is that of analysis of protein and moisture in wheat, where a bulk sample is
introduced to the instrument and is analysed many times over to produce a single averaged
predicted value, essentially following the same procedure, but can this procedure be
generalised? An appropriate answer would be related to how representative the reference
sample is with which the spectral replication/averaging approach is to be linked in a
calibration, in addition to the specific spot size issue pertaining to the instrument. There is
never analysis without (some form of) preceding sampling, sub-sampling, sample preparation
etc.
†† A full account of these interrelations can be found in Esbensen and Wagner [7].
10. An introduction to multivariate
classification
Figure 10.4: Dendrogram of clusters after application of HCA to NIR spectra of raw materials.
Figure 10.5: Cluster analysis of NIR spectra of raw materials using PCA.
Versicolor 48 2 0 96
Virginica 14 36 0 72
Setosa 0 0 50 100
Figure 10.7: 3D Scatter Plot of Fisher’s Iris data grouped by k-Means clustering.
Figure 10.8: Separation of classes of Iris species using HCA and Euclidean distance.
Figure 10.12: Each data class from Figure 10.10 as modelled by an individual (local)
PCmodel (SIMCA).
Classification results
The typical result output for a SIMCA model is the Classification
Table, which shows all objects and their classified, i.e. assigned class
memberships.
For each object, an asterisk is shown in the respective class
column for a particular model whenever the object in question has
been assigned as belonging to this highlighted model with the current
significance, i.e. when the object simultaneously satisfies both the Si
and Hi limits set (these statistics are described in more detail below).
Non-marked objects do not belong to any of the tested classes, i.e.
they will not contain an asterisk in any column of the Classification
Table. This is one of the most powerful aspects of SIMCA with
respect to many other of classification methods, its ability to identify
an object into “the null class”. This is a further development of
SIMCA’s power to deal with the asymmetric classification scenario
(also see further below).
Purpose
This plot shows the transverse (orthogonal) distance from all new
objects to two selected models (classes) at the same time (class pairs
can be treated in a sequential manner until all classes in a library have
been tested). The critical, cut-off class membership limits (S0)
constitutes the threshold for whether to assign objects as within or
outside the two classes in question. The statistical significance level
to be used in the classifications must be stated a priori otherwise the
assessment will be subjective and therefore biased. The higher the
significance level, the more strictly the new objects will be judged
with respect to “true” membership. This means that only “certain”
cases will be recognised as belonging to the “nearest class”; the
“doubtful” cases will fall outside. As part of the local model building, a
particular S0 limit may occasionally be changed by the data analyst by
changing the significance level, but importantly only for exploratory
purposes.
Figure 10.13: Typical classification situations when using the method of SIMCA.
Interpretation
Purpose
The Si vs Hi plot can also be called a membership plot w.r.t. one
selected model because it encompasses both the transverse as well
as the longitudinal types of limits used in the classification, i.e. the
distance to the model (the residual standard deviation) and of the
leverage (distance to model centre) measures for each object.
Objects that fall inside these limits are very likely to belong to the
class (model) at the chosen significance level. Alternative local
models can be tested for in succession.
This plot is similar to the Influence plot, used to detect outliers in
PCA and discussed in chapter 6. See below for an important caveat.
Interpretation
The Si vs Hi plot shows the object-to-model distance and the
leverage for each new object. The leverage can be gauged from the
abscissa and the distance from the ordinate. The class limits are
shown as horizontal threshold lines for the object-to-model distance
and similar vertical demarcations for the leverage limit. The limit for
the object-to-model distance again depends on the significance level
chosen. The leverage limit also depends on the number of PCs used
in the local model and is fixed for a given classification. The leverage
value shows the distance from each object to the model centre. In
one sense, the leverage tries to summarise the complete information
contained in the model, i.e. the variation described by the number of
PCs developed. It pays to remember well that both the Influence plot
(PCA) and the Si vs Hi plot (SIMCA) are but an attempt to simplify
(with a two-dimensional plot) the full multivariate relationships of
original dimensions (N, p). There is always a projection loss that must
be taken into account by the experienced multivariate data analyst—
who should always be looking at the data structures modelled in the
higher-order component space, just to be sure that no important
structural elements could be hiding here.
Figure 10.16 provides a generic Si vs Hi plot and its interpretation.
Objects near the origin, within both limits, are classified as bona
fide members of the model. Objects outside these lines are classified
as not belonging to the model. The further away from the origin of
the plot objects lie, the more different they are from the model
ensemble. Figure 10.17 shows the Si vs Hi plot for the model Setosa
when a test set of all classes was applied to the SIMCA model for the
Fisher Iris data. Objects close to the abscissa have short distances to
the model and in this case, represent Setosa, however, in some
cases, these may be extreme (they may well have large leverages,
reflected along the ordinate). Objects in the upper-right quadrant (e.g.
the Versicolor and Virginica objects in Figure 10.17) do not belong to
the class Setosa, while all Setosa objects lie well within all limits of
the test. Class membership characteristics such as within/outside a
model and “extreme objects” (objects that may in fact not belong) are
all specified by the significance level chosen. Extreme objects should
be checked based on domain-specific knowledge pertaining to the
problem context.
The above constitute all the information available from SIMCA
classification. It is now up to the data analyst to decide how to deal
with the specific information, i.e. how to view specific objects,
especially objects that lie close to the appropriate threshold limit etc.
Statistically, these objects are either inside or outside because of the
a priori significance level choice—however, there may be internal data
structures revealed at first by such comprehensive plots that may
occasionally lead to a more refined local PCA modelling (in many
situations these internal irregularities are also to be observed already
in the first in toto PCA projections).
Purpose
The Si/S0 vs Hi plot is quite similar to the Si vs Hi plot, above, and is
used the same way. However, the values of the ordinate axis in the Si
vs Hi plot are in absolute values. In the present Si/S0 vs Hi plot, the
object distance is expressed relative to the representative average
distance for the model (S0), thereby making it easier to relate the
transverse distance measures to one another.
Interpretation
The plot is interpreted in exactly the same way as the Si vs Hi plot, i.e.
objects in the lower-left corner belong to the class in question within
all pre-set limits etc. The Si/S0 vs Hi plot for the Setosa model is
displayed in Figure 10.18.
Purpose
The model distance plot is used to visualise the distance between
models, i.e. to quantify if they are really different. A large inter-model
distance indicates clearly separated models. The model distance is
found by first fitting objects from two given classes to their own
model as well as to the alternative model, and this double
classification scheme can be carried out on all data class models in
the library in turn. The model distance measures can then be
calculated from pooled residual standard deviations.
Interpretation
A useful rule-of-thumb states that a model distance greater than 3
indicates models which are significantly different. A model distance
close to 1 suggests that the two models are virtually identical (with
respect to the given data). The distance from a model to itself is of
course, by definition, 1.0. In the distance range 1–3 models overlap to
some degree.
The example in Figure 10.19 is the model distance plot for the
model Versicolor. Using the four measured variables, it was found
that two of the species are very similar. This is reflected in the model
distance plot where the distance from model Versicolor is shown. The
distance to the first model (Setosa) is large (around 50), but the
distance to the last model (Virginica) is smaller, and is a reason why
there is overlap of the models. The second bar in Figure 10.19 is the
distance to the Versicolor model itself, i.e. 1.0.
Purpose
A measure analogous to the above model distance, but calculated
from one model to all other alternatives can be plotted for the
individual variables. The discrimination power of a variable thus gives
information about its ability to discriminate between any two models.
The discrimination power is calculated from the residuals obtained
from fitting all objects from one model to all the alternative models—
compared to fitting to the specific class model.
If a poor classification is observed, deleting the variables with
both a low discrimination power and a low modelling power may
sometimes help. The rationale for this specific deletion is, of course,
justified by the fact that variables which do not partake in either the
data structure modelling nor in the inter-class discrimination are not
interesting variables—at least not in a classification perspective (they
may be otherwise interesting of course). However, there is a warning
here. If multiple classes make up a model, the Discrimination Power
is a pairwise test. If unimportant variables for the two models being
compared are removed, this has to be done for all models in the
library. N.B. unimportant variables for one pairwise comparison may
be important for another. It is therefore important that the data
analyst ensures that for the sake of one class comparison, the
integrity of the rest of the library is not compromised, i.e. if
unimportant variables from one class are eliminated to improve its
classification ability, this may be detrimental to the other classes in
the library (as they may be important variables for those classes). In
such cases, the Hierarchical Models discussed in chapter 13 may be
a better option.
Figure 10.18: The relative distance-leverage plot, Si/S0 vs Hi (same data as in Figure 10.17).
Interpretation
The discrimination power plot shows the discrimination power of
each variable in a given two-model comparison. A value near 1
indicates no discrimination power at all, while a high value, i.e. >3,
indicates good discrimination for a particular variable.
The example in Figure 10.20 again shows a plot from the Iris
species classification. Here the objects belonging to the Iris setosa
class are projected onto model Iris versicolor. The plot shows which
variables are the most important in discriminating between these two
classes. All the variables have a discrimination power larger than 3
and all are therefore useful in the overall classification.
Figure 10.19: Model distances for the three classical Iris classes.
Figure 10.20: Discrimination power plot for the four classical Iris variables.
Purpose
The modelling power plot is used to quantify the relevance of a
particular variable in the modelling of the individual local models. The
modelling power provides information on how much of the variable’s
variance is used to describe a particular class.
The modelling power can thus be a useful tool for improving an
individual class model. Even with careful variable selection, some
variables may still contain little or no information about the specific
class properties. Thus, these variables may have a different variation
pattern from the others, and consequently, they may cause the model
to deteriorate. Different variables may show different modelling power
in different models, however, so one must again keep a strict
perspective with respect to the overall classification objective(s) when
dealing with multi-class problems.
Variables with large modelling powers have a large influence on
the model. If the modelling power is low, i.e. below 0.3, the variable
may make the model worse and therefore be a candidate for deletion,
particularly if its discrimination power is also low.
Interpretation
The modelling power is always between 0 and 1. A rule-of-thumb is
that variables with a value equal to or lower than 0.3 are less
important.
In Figure 10.21 the modelling powers for the four Iris variables are
shown w.r.t. the Setosa model. All variables are important for
modelling the class Setosa. It is not a tradition in chemometrics to try
to polish every variable set to the limit, indeed very many times low
modelling variables can be kept without loss of information and this is
where subject matter expertise is paramount.
X-variables Y-variables
5 88 1.98 0.57 –1 1 –1 B
… … … … … … … … …
a
The samples in the data set may or may not be sorted by class; this
is without relevance for the model, but will of course help make the
overview of more complex cases clearer. N.B. The largest number of classes
treated with PLS-DA by any of the authors of the present book is 7.
Figure 10.22: Defining statistical limits around target values in PLS-DA for class rejection.
Figure 10.25: Using a Kernel Function to map a high-dimensional, complex space into a
simpler, linearly separable space.
As is the case with any multivariate data analysis, start simple and
then build up into more complex solutions only when/if the simpler
methods do not show promise. To aid in the selection of the
parameters used for fine tuning the class boundaries in SVM, many
software applications provide a grid search approach in which
various combinations of parameters are assessed to show the quality
of the final discrimination solution. When evaluating kernel
parameters, always use the smallest numbers possible to avoid
overfitting. As the linear function is not always able to model complex
separation problems, data are sometimes mapped into a new feature
space and a dual representation is used with the data objects
represented by their dot product.
After mapping the original data space into the simpler Kernel
space, the SVMC algorithm will search for the objects that lie on the
borderline between the classes, i.e. to find the objects that are ideal
for separating the classes; these objects are named Support Vectors.
The support vector is defined as the reduced training data from the
kernel.
Figure 10.26 shows this principle where objects marked with + for
the two classes are used to generate the rule for classifying new
objects.
Unlike LDA and SIMCA, where the statistical boundaries are
defined by the standard deviation (or manifestations thereof) of the
samples from the class centres or the local PCA models, SVMC does
not take into account unequal class variances that can result in class
overlap and potential ambiguity between classes, but only takes into
account important objects close to and along the class separation
boundaries, the important support vectors.
To assess the quality of the separation, the SVMC also
establishes margins around the line of best separation (these are the
lines labelled H in Figure 10.26). If the margins are clear of any
objects, then the separator is said to be of “high quality” and is
suitable to discriminate between classes. When objects still exist in
the margins, then there is the likelihood of ambiguities when the
SVMC model is applied to new data.
SVM is a classification method based on statistical learning
wherein a function that describes a hyperplane for optimal separation
of classes is determined. It has advantages over other classification
methods such as neural networks, as its outputs are more
transparent, and has less tendency of overfitting when compared to
other non-linear classification methodologies.
For these more complex modelling algorithms, which are based
on more special assumptions, model validation is progressively the
critical success factor to avoid the dangerous likelihood of overfitting.
In the case of SVMC, a few objects are singled out for the crucial
definition of margins—these better be beyond reproach regarding
representativity! However, SVMs are particularly effective for the
modelling of non-linear data, and they are also relatively insensitive to
variation in the model parameters. SVM uses an iterative training
algorithm to achieve separation of different classes. In the case of
similar classification rates for the training data, the model with the
most linear parameters and minimum number of support vectors
should be chosen.
Figure 10.26: Separation of data using Support Vectors and definition of margins.
Using the method of PCA on the data shown in Figure 10.28 resulted
in the PCA overview of Figure 10.29.
The main features of the PCA overview are listed as follows:
1) Two PCs describes 99% of the total data variability.
2) The score plot shows distinct clusters of the oil varieties.
3) The loading plots for PCs 1 and 2 show some similar and some
distinct chemical features along both PCs.
4) The influence plot indicates that the corn margarine samples are
extreme. This is because they are the most different and least
represented samples in the training set.
This PCA now justifies that the FTIR method is capable of
classifying oil varieties of the four classes under investigation. The
process of Supervised Classification can now begin.
Figure 10.28: SNV transformed FTIR data of vegetable oils using a reduced spectral region.
For each class present in the PCA score plot, an individual PCA
model was developed and validated for use as a classification rule.
These models were compiled into a library for use as a SIMCA
classification routine in The Unscrambler®. The test set comprised of
samples from the four classes investigated and some negative
challenge samples of the following oil types:
1) Walnut Oil,
2) Peanut Oil,
3) Sesame Oil,
4) Soybean Oil.
Negative challenge samples are always suggested when testing a
classification method as they assess the libraries discrimating ability
of classes not present in the library. If these samples positive identify
as a known class in the library, then the library is not sensitive enough
to distinguish between some classes.The SIMCA library was applied
with a significance level set to 5%. The classification table is shown in
Figure 10.30 for this analysis.
The main features to note from the Classification Table are the
following:
1) In no cases were there more than one asterisk per row. This
indicates that there were no ambiguous classification results
obtained.
2) In each case, the sample in the test set was correctly classified as
the class it belonged to.
3) There are, however, some samples that did not classify into their
own class. This is an indication that the training model does not
have all of the variability it needs to be robust. This will be
confirmed using model diagnostics.
4) Of the samples that had no library available, Soybean Oil classified
as Corn Oil. This result indicates to the developer that in the future,
when developing a Soybean Oil classification model, there could
be ambiguities with Corn Oil. The topic of Hierarchical Models is
discussed in chapter 13 section 13.6.
Figure 10.29: PCA overview of SNV pre-processed vegetable oil FTIR spectra.
The first model diagnostics plot to view is the Coomans’ plot. This is
shown in Figure 10.31 for the closest groups from the original PCA,
being Safflower and Corn oil (refer to the scores plot in Figure 10.29).
Coomans’ plot reveals the following information based on the
pairwise comparison of Corn and Safflower oils:
1) Since there are no Corn and Safflower Oil samples in the region
bounded by the model limits and the origin, there is no suggestion
that these oils cannot be separated from each other.
2) In the case of Corn Oil, when a Soybean Oil model is to be
considered, then it may result in some ambiguity issues, as already
detected in the Classification Table.
3) All other Oil types are distinctly different from Corn and Safflower
Oils as evidenced by them lying outside the model limit bounds, i.e.
in the top-right quadrant.
Similar results to these were found for all pairwise oil class
comparisons. Figure 10.32 shows the Si vs Hi plot for the oil type
Olive.
Figure 10.32 is focused on the Olive Oil samples only and it shows
that the nearest class to it is Peanut Oil. It is now apparent why one
of the samples did not classify for Olive Oil in the Classification Table
of Figure 10.30. One sample has a greater leverage than the model
limit at 5% significance. This suggests that more samples are
required to make the model robust. If this sample lay outside both
model limits, if could be justified as being an outlier and would be
investigated in a root cause analysis.
Figure 10.30: SIMCA Classification Table for Test Set of vegetable oils scanned by FTIR
spectroscopy.
Figure 10.34: PCA-LDA results for Vegetable Oil classification by FTIR with the confusion
matrix for the analysis.
The results in Table 10.4 show that for the positive tests, the PCA-
LDA model works well, but when it is applied to the sample types not
included in the model, it tries to fit these samples to the existing
classes (which is fundamentally inappropriate for a reliable
classification method).
Figure 10.35: Grid search dialog for SVMC as implemented in The Unscrambler®.
The three other methods used, PLS-DA, PCA-LDA and SVMC all
make use of single-step training models (with SVMC requiring a
method choice, a Kernel and optimisation of model parameters).
Although the three methods resulted in no misclassifications of test
samples from the class models used to develop the library, these
methods fundamentally failed when challenged by samples not
belonging to the library.
Table 10.5: Confusion Matrix for vegetable oil SVMC FT-IR data.
Corn 9 0 0 0
Olive 0 15 0 0
Safflower 0 0 11 0
Corn Marg 0 0 0 6
10.11 References
[1] Miller, J.N. and Miller, J.C. (2005). Statistics and Chemometrics
for Analytical Chemistry, 5th Edn. Prentice Hall.
[2] Adams, M.J. (1995). Chemometrics in Analytical Spectrosopy,
Ed by Barnett, N.W. RSC Analytical Spectroscopy Monographs.
[3] Everitt, B.S., Landau, S. and Leese, M. (2001). Cluster Analysis,
4th Edn. John Wiley & Sons Inc.
https://doi.org/10.1201/9781420057492.ch10
[4] Romersburg. H.C. (1984). Cluster Analysis for Researchers.
Lifetime Learning Publications, Belmont, California. Reprint
edition Feb 1990, Krieger Publishing Company.
[5] Fisher, R.A. (1936). “The use of multiple measurements in
taxonomic problems”, Ann. Eugenics 7, 179–188.
https://doi.org/10.1111/j.1469-1809.1936.tb02137.x
[6] Wold, S. (1976). “Pattern recognition by means of disjoint
principal components models”, Pattern Recogn. 8, 127–139.
https://doi.org/10.1016/0031-3203(76)90014-5
[7] Massart, D.L., Kaufman, L. and Esbensen, K.H. (1982).
“Hierarchical nonhierarchical clustering strategy and application
to classification of iron meteorites according to their trace
element patterns”, Anal. Chem. 54(6), 911–917.
https://doi.org/10.1021/ac00243a017
[8] Esbensen, K.H., Kaufmann, L and Massart, D.L. (1984).
“Interobject structure of ungrouped iron meteorites as revealed
by advanced clustering: Methods and chemical features”,
Meteorit. Planet. Sci. 19(2), 95–109.
https://doi.org/10.1111/j.1945-5100.1984.tb00032.x
[9] Eriksson, L., Johansson, E., Kettaneh-Wold, N., Trygg, J.,
Wikstrøm, C. and Wold, S. (2006). Multi- and Megavariate Data
Analysis Part I: Basic Principles and Applications. Umetrics Inc.,
Umeå, Sweden.
[10] Brereton, R.G. and Loyd, G.R. (2014). “Partial least squares
discriminant analysis—taking the magic away”, J. Chemometr.
28, 213–225. https://doi.org/10.1002/cem.2609
[11] Naes, T., Issakson, T., Fearn, T. and Davies, T. (2017). A User
Friendly Guide to Multivariate Calibration and Classification. IM
Publications, Chichester, UK.
[12] Luts, J., Ojeda, F., Van de Plas, R., De Moor, B., Van Huffel, S.
and Suykens, J.A.K. (2010). “A tutorial on support vector
machine-based methods for classification problems in
chemometrics”, Anal. Chim. Acta 665, 129–145.
https://doi.org/10.1016/j.aca.2010.03.030
[13] Vong, R., Geladi, P., Wold, S. and Esbensen, K. (1988). “Source
contributions to ambient aerosol calculated by discriminant
partial least squares regression (PLS)”, J. Chemometr. 2, 281–
296. https://doi.org/10.1002/cem.1180020406
* Yes, you read correctly: In the presence of a significant noise component in the data, the C
is to be fixed by a cross-validation procedure. Based on comprehension of the
representativity issues detailed in chapters 3 and 8 (see also chapter 9), one should feel
suitably concerned placing this critical parameter on the vagaries of this mandated validation
approach.
Chapter 11. Introduction to Design of
Experiment (DoE) Methodology
Figure 11.1: Volume vs resting time. First set of experiments, with the traditional yeast, 15g
per kg of flour and a temperature of 37°C.
Figure 11.3: Optimum in response surface. Contour plot showing the values of bread volume
for all possible combinations of resting time and amount of yeast.
A Full Factorial Design, is the easiest type to describe how DoE works
in practice. Consider three factors: the amounts of salt (factor 1),
sugar (factor 2) and lemon (factor 3) used in a sports drink recipe.
Each factor is varied from a low (–) to a high (+) level based on a
simple tree diagram (refer to Figure 11.4).
Starting from the origin, factor 1 (salt) is varied at a high (+) level
and a low (–) level. This results in two experimental runs. The next
factor (sugar) is added to the design. Now, for each level of salt, a
high and low level of sugar must be run. This results in four
experimental runs. Finally, for the third factor (lemon) for each run of
combinations of salt and sugar, a high and a low level of lemon must
be run. This results in eight experimental runs in total.
Figure 11.4: Simple tree diagram for constructing a full factorial design.
Effects
i.e. by finding the regression coefficients bi. This can be done using
several methods. MLR is the most usual, but PLSR or PCR (chapter
7) may also be used. If there are three design variables and it is
desired to investigate one response, the following equation will be
used (equation 11.2).
Effects
1 1 –1 –1 –1 1 1 1 –1 1.7
a 1 1 –1 –1 –1 –1 1 1 2.1
b 1 –1 1 –1 –1 1 –1 1 5.2
ab 1 1 1 –1 1 –1 –1 –1 2.8
c 1 –1 –1 1 1 –1 –1 1 4.5
ac 1 1 –1 1 –1 1 –1 –1 3.5
bc 1 –1 1 1 –1 –1 1 –1 7.2
abc 1 1 1 1 1 1 1 1 4.8
There are eight terms in the above equation. The first term b0 is
the intercept of the model and has a physical value of the mean of all
of the response terms.
The main effects associated with any factor are the average of the
observed difference of the response when that factor is varied from
the low to the high level. The estimated effect equals twice the b-
coefficient for the factor in the regression equation. This is because
the design is run at two levels and by multiplying the coefficient by +1
and –1, this results in a doubling of the range. For the 23 design, there
are three main effects associated with the factors.
An interaction effect for example, AB means that the influence of
changing variable A will depend on the setting of variable B. This is
analysed by comparing the effects of A when B is at different levels. If
these effects are equal, then there is no interaction effect AB. If they
are different, then there is an interaction effect. Estimated interaction
effects again equal twice the corresponding b-coefficients. For the 23
design, there are three two-factor interactions (2FI) and one three-
factor interaction (3FI). In the case where there is no interaction
between variables, when an effects plot is drawn, the lines from low
to high (or vice versa) will be parallel to each other.
In general, to estimate how many main effects and interactions
there are in a factorial design, the use of Pascal’s Triangle is required.
This is shown in Figure 11.10 for designs up to five factors.
Taking the five-factor experiment as an example, the following
main effects and interactions are obtained,
1 × Intercept Term
5 × Main Effects
10 × 2FIs
10 × 3FIs
5 × 4FIs
1 × 5FI13
Pascal’s Triangle shows that for the 25 full factorial design 16 of
the 32 experiments are used to determine three-factor or higher
interactions. The next section on Fractional Factorial designs
provides methods for reducing the number of experiments in the
presence of high order interactions.
11.2.10 Confounding
The price to be paid for performing fewer experiments is called
confounding, which means that some terms will not be able to be
distinguished from each other. This happens because of the way the
fractional designs are built; using some of the resources that would
otherwise have been devoted to the study of interactions, these are
now used to study main effects of more variables instead.
Confounding is sometimes also referred to as Aliasing.
a +1 –1 –1
b –1 +1 –1
c –1 –1 +1
abc +1 +1 +1
1 + – – – – + + + + + + – – – – +
a + + – – – – – – + + + + + + – –
b + – + – – – + + – – + + + – + –
ab + + + – – + – – – – + – – + + +
c + – – + – + – + – + – + – + + –
ac + + – + – – + – – + – – + – + +
bc + – + + – – – + + – – – + + – +
abc + + + + – + + – + – – + – – – –
d + – – – + + + – + – – – + + + –
ad + + – – + – – + + – – + – – + +
bd + – + – + – + – – + – + – + – +
abd + + + – + + – + – + – – + – – –
cd + – – + + + – – – – + + + – – +
acd + + – + + – + + – – + – – + – –
bcd + – + + + – – – + + + – – – + –
abcd + + + + + + + + + + + + + + + +
1 + – – – – + + + + + + – – – – +
ab + + + – – + – – – – + – – + + +
ac + + – + – – + – – + – – + – + +
bc + – + + – – – + + – – – + + – +
ad + + – – + – – + + – – + – – + +
bd + – + – + – + – – + – + – + – +
cd + – – + + + – – – – + + + – – +
abcd + + + + + + + + + + + + + + + +
X4 = X1X2X3 X1X2X3X4 = I
11.2.11 Types of variables encountered in DoE
Figure 11.13: Centre samples for the 22 design with one categorical factor.
Setting the ranges of variation for all factors in the design is a critical
setup that will govern the success of the experimentation. In all
cases, make the range of variation large enough to induce an effect
and small enough to be realistic. If it is suspected that two of the
designed experimental runs will give extreme, opposite results,
perform those runs first (although this breaks the principles of random
sampling, it makes practical sense to do this). If the two results are
indeed different from each other, this means that enough variation
has been generated. If they are too far apart, too much variation has
been generated, and reduction of the ranges must be considered. If
they are too close, try a centre sample: this will indicate the presence
of a very strong curvature!
Figure 11.14: Centre samples for the 23 design with one categorical factor.
Figure 11.15: Centre samples for the 23 design with two categorical factors.
11.2.13 Replicates
11.2.14 Randomisation
Screening designs
At the start of a new project, there is usually a large number of
potentially important variables to consider. At this stage, the aim of
any experimentation is to find out which are the most important
variables. For a first screening, the most important rule is: do not
leave out a variable that might have an influence on the responses,
unless it is known beforehand that it cannot be controlled in practice.
It would be costlier to have to include one more variable at a later
stage, than to include one more into the first screening design.
For a more extensive screening, variables that are known not to
interact with other variables can be left out. If those variables have a
negligible linear effect, these variables can be set to a constant value
(like for instance: the least expensive). If those variables have a
significant linear effect, then they should be fixed to the most suitable
level to get the desired effect on the response.
1 + – – – + + + –
a + + – – – – + +
b + – + – – + – +
ab + + + – + – – –
c + – – + + – – +
ac + + – + – + – –
bc + – + + – – + –
abc + + + + + + + +
1 × Intercept Term
10 × Main Effects
45 × 2FIs
120 × 3FIs
210 × 4FIs
252 × 5FIs
210 × 6FIs
120 × 7FIS
45 × 8FIs
10 × 9FIs
1 × 10FIs
Design redundancy
One of the biggest advantages of DoE over other approaches to
experimentation is known as Design Redundancy. If an experimenter
takes a risk by using a low resolution fractional factorial design, the
more factors that can be eliminated as insignificant, the more
powerful the design becomes. To illustrate design redundancy, the 23
design of main effects only will be used. This is shown in Table 11.6.
Looking at column X3, the first 4 rows are (–) signs and the last
four rows are (+) signs. Now, looking at column X2, the first 4 rows
consist of 2 (–) signs and 2 (+) signs all carried out at X3 at its (–) level.
The same pattern of (–) and (+) signs is repeated in column X2 for X3 at
its (+) level, i.e. X2 is replicated for the levels of X3 present. But what if
X3 is insignificant? If a factor is found to be insignificant, it can be
removed from the design, thus the design table now looks like the
one shown in Table 11.7.
When X3 (or for that matter any factor) can be removed from a
design, the resulting table is replicated and can be analysed without
the insignificant factor(s), thus providing more confidence in the
precision of the measurements. Taking the example further, if any of
X1 or X2 is also found to be insignificant, then the resulting design
table is replicated twice.
Relating this back to a low-resolution screening design, the same
principles as above hold when insignificant factors can be removed
from the design. For example, if a MSA = SSA / νA design is selected,
this assesses seven factors in eight experiments. Now if three factors
are found to be insignificant, then the resulting design becomes four
factors in eight experiments, or the design is now the half fraction
design, where main effects can be assessed and 2FIs are
confounded with each other. In this case, should there be a need to
resolve any 2FI confounding patterns, the design can be extended by
performing the alternate fraction of the half fraction design.
Term X1 X2 X3
1 – – –
a + – –
b – + –
ab + + –
c – – +
ac + – +
bc – + +
abc + + +
Term X1 X2
1 – –
a + –
b – +
ab + +
c – –
ac + –
bc – +
abc + +
Plackett–Burman designs
Optimisation designs
Once the majority of important variables have been isolated and
analysed, the next step in the DoE process is usually to find the
“sweet spot” of the process/product. The purpose of an optimisation
design is to investigate the remaining variables at more than two
levels, so that more complex models, which account for curvature,
can reveal the nature and stability of the optimal point.
Since the purpose of such designs is to study what happens
anywhere within a given range of variation, optimisation designs can
only investigate factors which vary over a continuous range. As a
consequence, if any categorical variables have been found to be
significant, in the optimisation design, the best level of the variable
will be chosen at the optimisation stage.
Optimisation designs consist of a set of experiments which study
at least three levels of the factors in a balanced way. The two most
common approaches to optimisation designs are,
Figure 11.17: The principle of steepest ascent.
Design Total
variables Cube Star Centre runs
2 4 4 3–5 11–13
3 8 6 3–6 17–20
4 16 8 3–7 27–31
5 32 10 3–10 45–52
6 64 12 3–15 79–91
Figure 11.19: Central composite design where low star = low cube, high star = high cube.
Box–Behnken designs
Design Total
variables “Cube” Centre Runs
3 12 5 17
4 24 5 29
5 40 3–6 43–46
6 48 3–6 51–54
In the case where the factors are limited to reside within a cube but
a rotatable design is desired, the Box–Behnken design is the only
one with that combination of properties.
The Box–Behnken design is more economical than the Central
Composite Design in terms of experiments that have to be
performed.
In the case where a factorial design was first constructed, but
provision was not previously made to extend to an optimisation
design (due to a lower boundary on one or more factors) then a
Face Centred Cubic (FCC) design may be the only choice available.
where SSTotal = Total Sum of Squares, the total variability that can be
explained in the data set; SSReg = Regression Sum of Squares, the
proportion of the total variation described by fitting a model to the
data; and SSError = Residual Sum of Squares, the variability remaining
after fitting a regression model.
In general, the total variability must be partitioned over what can
be explained in the data (SSReg) and what cannot be explained by the
model (SSError). Significant models have a much higher value of SSReg
compared to SSError.
Since a regression model is made up of the individual regressors
(i.e. the effects of the experimental factors), SSReg is further broken
down into terms for each effect in the ANOVA table. To test the
significance of a particular effect, the response’s variance accounted
for by that effect must be compared to the residual variance which
summarises experimental error. If the “structured” variance (due to
the effect) is no larger than the “random” variance (error), then the
effect can be considered negligible. Otherwise it is regarded as
significant.
In practice, the generation and analysis of an ANOVA table is
achieved through a series of successive computations.
First, several sources of variation are defined. For instance, if the
purpose of the ANOVA model is to study the main effects of all
factors, each factor is a source of variation. Experimental error is
also a source of variation.
Each source of variation has a limited number of independent ways
to cause variation in the data. This number is called the degrees of
freedom (ν).
Response variation associated to a specific source is measured by
a Sum of Squares (SS).
Response variance associated to the same source is then
computed by dividing the sum of squares by the number of
degrees of freedom. This ratio is called Mean Square (MS).
Once mean squares have been determined for all sources of
variation, F-ratios associated to every tested effect are computed
as the ratio of MSeffect to MSerror. These ratios, which compare
structured variance to residual variance, have a statistical
distribution which is used for significance testing. The higher the
ratio, the more important the effect.
Under the null hypothesis (chapter 2) that an effect’s true value is
zero, the F-ratio has a Fisher distribution. This makes it possible to
estimate the probability of getting such a high F-ratio under the null
hypothesis. This probability is called the p-value; the smaller the p-
value, the more likely it is that the observed effect is not due to
chance. Usually, an effect is declared significant if the p-value <
0.05 (significant at the 5% level). Other classical thresholds are
0.01 and 0.005.
The ANOVA results are traditionally presented in the format shown
in Table 11.10.
The calculations associated with the ANOVA table are discussed
in later sections of this chapter, however, before these calculations
can be described, a discussion on important effects is provided in the
next section.
The Pareto Chart shows the t-values of each effect as a bar plot.
Figure 11.22 provides an example Pareto Chart for the Sports Drink
data provided in Table 11.1. In this chart, the t-value limit for the
effects is shown as the lower limit and the Bonferroni Limit (a family-
wise adjustment to the t-values for taking into account the
dependency of the t-test for multiple variables) is plotted as the upper
limit.
Effects that lie between the two limits should be tested for
significance when selected for addition to the model. In fact, effects
that exceed the Bonferroni limit are almost certainly significant. In the
case of the Sports Drink example, the response is sweetness and
from the Pareto Chart, Sugar is the most contributing factor to
sweetness. Even to a non-expert, this outcome makes sense.
Furthermore, all three main effects and the Salt × Sugar
interaction contribute to sweetness. All other interaction terms do not
contribute to the model. This means that only the ingredients Salt and
Sugar antagonistically combine to enhance flavour (since the AB
interaction term is negative).
C νc = # levels SSC
–1
In the above example for the Sports Drink analysis, all main effects
and one interaction term were found to be significant, but what if a
significant interaction term was found which contains an insignificant
main effect?
If a significant interaction term contains an insignificant main
effect, the main effect must also be included in the model so as to
calculate the significant interaction term. This is called Hierarchy of
Effects and programs such as Design-Expert® will always include
insignificant main effects into the model when they take part in a
selected significant interaction.
where b0 is the intercept (or mean response) of the model; bi are the
regression coefficients to be estimated by least squares for each
factor in the model; Xi are the main terms selected in the model; and ɛ
is the residual term (i.e. what the model cannot explain).
where all terms are the same as for the linear model, except for, bij
are the regression coefficients to be estimated by least squares for
each interaction term and XiXj are the interaction terms selected in the
model.
Note: In the interaction terms, i ≠ j (in the case of the 3FI, i ≠ j ≠ k)
The following sections describe the general calculations involved
in generating the ANOVA table.
SStotal is the total variation around the mean response that can be
explained by a model. The larger SStotal is, the more likely a significant
model will result. If SStotal is small, this may be an indication that,
1) The range of the factors used was not large enough to induce a
detectable response or
2) The factors used contain no information related to the response.
SStotal is calculated using equation 11.4,
SSReg is the variation described by the fitted model and accounts for a
specific proportion of SStotal. The nature of the fitted model can
change, based on the number of terms added to the equation. In this
case, each term in the model accounts for a specific proportion of
SSReg. The form of the model is determined by the design and the fit
of the model to the data. In particular,
1) The form of a screening design is linear as only the most significant
main effects are sought.
2) The form of a factor influence design is either purely linear or linear
with interaction terms.
3) The form of an optimisation design is anything from linear to quartic
in nature.
SSReg is calculated using equation 11.5,
This process is performed for all terms used in the fitted model
and the following balance defined in equation 11.7 should exist if all
calculations are performed correctly,
Excluding the intercept term, there are seven model terms and
each take up one degree of freedom. This results in a saturated
design, i.e. there are no degrees of freedom left to estimate the model
error term. So, how can more degrees of freedom be generated?
There are three options available to the experimenter,
1) Replicate the entire design: For small designs (i.e. k ≤ 4) this may
be possible, however, for large designs, or where there is cost
sensitivity, this option may not be a viable solution. For the 23
design, a full replicate will result in 16 experiments, with 15 total
degrees of freedom and 15 – 7 = 8 degrees of freedom to estimate
the error contribution.
2) Add Centre Points: As described earlier in section 11.2.11, centre
points do not contribute to calculating model terms in factorial
designs, but serve the dual purpose of allowing the detection of
model curvature and providing degrees of freedom for calculating
error terms in saturated designs. In the case of the 23 design,
adding three centre points provides 3 – 1 = 2 degrees of freedom
for estimating model error.
3) Removing insignificant model terms: When a model term is found
to be insignificant, it does not contribute to estimating SSReg. In this
case, by removing the term from the model, the model becomes
simpler and the excluded term has to be partitioned into SSError.
This is the simplest way to add degrees of freedom to a saturated
model as it does not require additional experimentation.
The only drawback to point 3 above occurs when every term
contributes to the model. In this case one of the approaches of points
1 and 2 can be performed to estimate model error and therefore
model significance. This is the topic of the next section.
The F-test
Fishers F-test was described in detail in chapter 2 for determining
whether one source of variance was significantly different from
another. In the case of DoE, the F-test is set up to determine whether
the variance described by the fitted model is significantly different
from the residual (i.e. the portion of the variance not described by the
model). This situation is defined by equation 11.9,
p-value
A complementary measure to the F-values is the p-value. The p-value
is the probability that the model or a particular model term = 0. For
example, pTerm A = 0.05 means that the effect of term A is significant
with a probability of 95%. p-values are best described with reference
to a histogram (see Figure 11.25).
If the p-value obtained is less than the significance level stated,
then the effect is considered to be significantly different from zero on
a statistical basis. Unfortunately, there are a few pitfalls that are
generally not known by many practitioners, these include,
The p-values are totally irrelevant if the number of degrees of
freedom for the estimate of the error is small, i.e. there were too
few replicates to estimate the error.
All p-values become low, i.e. all effects appear to be significant, if
the error is very small. This may occur if the reference method is
very accurate.
The un-replicated 23 Full Factorial design has seven model terms and
an intercept term. The model terms use all of the degrees of freedom
available (if all terms are used in the model). The sports drink example
(refer to Table 11.1) will be used to demonstrate the construction of
an ANOVA table.
In Figure 11.24, the half normal plot of effects showed that the
Salt, Sugar and Lemon main effects are important and the AB
interaction term was also important, the remaining terms were
statistically insignificant. This means that the final model is the
interaction model consisting of four terms and the other model terms
provide degrees of freedom for calculating SSError. The form of the
model is provided as follows,
SSTotal = 23.16
This value represents the total corrected sum of squares that can
be explained around the mean of the response value (corrected infers
that the data are mean centred). To calculate SSReg, equation 11.3 is
used and based on the normal probability plot, only four of the
available seven model terms are used to calculate this value,
SSReg = 22.68
Table 11.11: Model term sums of squares for the sports drink
example.
A 3.65
B 8.41
C 8.41
AB 2.21
SSError = 23.16 – 22.68 = 0.48
A 3.65 1 3.65
B 8.41 1 8.41
C 8.41 1 8.41
AB 2.21 1 2.21
where ei is the residual for sample i and hii is the leverage of sample
i.
R-Squared: This is the proportion of the model sum of squares
(SSReg) divided by the total sum of squares (SSTotal). The general
form of the expression is provided in equation 11.11.
As a general rule, R2Adj and R2Pred should be within 0.2 of each other.
If this is not the case, then an analyst is advised to check the data
for outliers, or revise the form/fit of the model.
Adequate Precision: This is a signal-to-noise ratio that compares
the range of the predicted values at the design points to the
average prediction error. As a general rule, ratios greater than 4
indicate adequate model discrimination.
where
Pressure 400 300 Below 300 psi, recirculation issues could occur
(psi) and above 400 psi, unnecessary wear and tear
on the pump may occur.
Stirring rate 250 150 Below 150 rpm, it is believed that not enough
(rpm) mixing energy is available to maintain the
reaction and above 250 puts unnecessary load
on the mixing impeller.
a 1 –1 –1 –1 –1 1 69.5
b –1 1 –1 –1 –1 1 48.6
c –1 –1 1 –1 –1 1 66.8
abc 1 1 1 –1 –1 1 64.1
d –1 –1 –1 1 –1 1 44.1
abd 1 1 –1 1 –1 1 99.5
acd 1 –1 1 1 –1 1 83.2
bcd –1 1 1 1 –1 1 68.6
1 –1 –1 –1 –1 1 2 45.9
ab 1 1 –1 –1 1 2 64.1
ac 1 –1 1 –1 1 2 59.5
bc –1 1 1 –1 1 2 77.7
ad 1 –1 –1 1 1 2 95.9
bd –1 1 –1 1 1 2 45.9
cd –1 –1 1 1 1 2 73.2
abcd 1 1 1 1 1 2 92.3
Figure 11.26: Half normal probability plot of effects, chemical reaction data.
Table 11.16: ANOVA table of the chemical process data with one
blocking variable.
Block 6.25 1 6
Error 155 9 17
Total 4736 15
1 71.2
68.1
65.3
2 65.2
67.3
69.9
Note: If any of the Block or Curvature terms are not required in the
calculation, they can be removed as necessary. In the case of no
blocks and curvature, the standard partitioning of sum of squares
between model and residuals holds.
Step 6: Residual degrees of freedom.
The residual degrees of freedom can now be calculated by difference,
It is important to note for equation 11.16, this only holds for centre
point calculations of pure error for non-blocked designs. For Pure
Error calculations involving replicated or blocked designs, the
interested reader is referred to the text by Myers and Montgomery [4].
SSLoF is the Sum of Squares due to the model not fitting its
intended form, i.e. if a linear model is fitted to quadratic data, then
this would result in an inflation in the error due to Lack of Fit (LoF).
The Sum of Squares due to Lack of Fit is calculated using equation
11.17.
Table 11.18: ANOVA table of the chemical process data with one
blocking variable and curvature.
Total 4769.0 21
Total 4769.0 21
νPE = nC – 1
Table 11.20: ANOVA table of the chemical process showing LoF and
PE.
PE 29.0 5 6.0
Total 4769.0 21
To date, the ANOVA table has been described as the main point of
the analysis of a designed experiment. There are a number of highly
important graphical tools also available to supplement the numerical
outputs of the ANOVA table. For the purpose of this text, the
following diagnostics tools will be described,
1) The Normal Probability plot of Residuals;
2) The Residual vs Predicted plot;
3) The Residual vs Run plot; and
4) The Predicted vs Actual plot.
To supplement the above diagnostics tools, there are a number of
model interpretation tools available. The following will be discussed in
more details in this section,
1) One-Factor and Interaction plots;
2) Contour and Response Surface plots; and
3) Cube plots.
Model diagnostics plots
Once normality (or near normality) of the model has been established,
the next step is to interpret the model, understand how it works and
finally put it into practice. Model interpretation plots form the basis of
what most people have come to expect from DoE. In particular, the
Response Surface plot is typically the flagship plot used to show how
DoE works. The following subsections discuss the various
interpretation plots and their usage.
After the significant main effects and interactions have been isolated
and the final model is built, the degree of change in various factors
may be investigated when moving from the low level to the high level
of that factor. The One-Factor plots in Design-Expert® allow an
analyst to investigate the influence each factor has on the response
variable, but most importantly, it allows the visualisation of how one
factor changes when other factors have been changed.
Figure 11.32: Predicted vs actual plot, chemical reaction data.
Using the Chemical Process data, the final model used factors
Temperature, pH and Stirring and the two-factor interactions of each
factor with Temperature. This means that if Temperature is to be
studied by itself, since it is influenced also by pH and Stirring, its
impact on Yield should also change as these other factors are
changed. Three scenarios are provided in Figure 11.33.
Since there are so many permutations only the effect of pH on
temperature is shown in Figure 11.33. In Figure 11.33a) the effect of
temperature is shown when factors pH and Stirring are at their centre
values. It shows that when all other factors are constant, the effect of
changing Temperature from a low to a high value has a positive
influence on yield.
Figure 11.33b) shows how temperature effects yield when pH is
moved to its low level (keeping stirring constant). This change in pH
has a detrimental effect on yield when temperature is at its low level,
but has little effect on yield when temperature is at its high level.
Figure 11.33c) shows how temperature effects yield when pH has
moved to its high level (keeping stirring constant). In this case, slightly
improved yields at low temperature are achieved compared to pH at
its centre point and when both temperature and pH are high, this
slightly reduces the yield.
One-Factor plots are typically useful when main effects are not
involved in significant interactions. Factors involving significant
interactions are best interpreted using the Interaction Plot.
The same information as shown in Figure 33 (a–c) can be
summarised in the Interaction Plot of Figure 11.34.
Figure 11.35 shows the two-factor interaction plot for the
Temperature Stirring interaction.
The interpretation of Figure 11.35 is as follows, when stirring is set
to its low value, its interaction with temperature is also low and there
is no real change in yield, however, when temperature and stirring are
at their high level, this has a marked synergistic effect where yield is
increased far greater than the two factors acting independently.
When the number of significant interactions becomes large, the
use of One-Factor and Interaction plots can become difficult and
sometimes confusing. What is required is a map that shows an
analyst the path to the optimal point. This is usually achieved with
Contour Plots and Response Surfaces.
Figure 11.33: One-factor plots for factor temperature, chemical reaction data.
Contour plots and response surfaces
These plots are the heart and soul of DoE. They provide a map, much
the same way as a mountaineer would use a map to navigate through
certain terrains, the Contour plots (also known as Response Surfaces)
are used to provide a “top down” view of the experimental space.
The size and shapes of the contours are dictated solely by the form of
the model used. In particular,
1) When there are no interactions between factors, the response
surface is a flat plane with straight contours.
2) When there are significant interaction terms, the plane becomes
twisted and the response surface starts to introduce curved
contours into the system.
3) When higher order models are used in Optimisation Experiments,
the response surfaces can become highly curved (since the models
are now based on quadratic, cubic or even quartic polynomials).
The response surface is a plot that shows how the response
variable changes over the experimental space studied for two
variables only. However, software packages such as Design-Expert®
allow the visualisation of response surfaces for the two variables
plotted and show how these change when other variables that are not
plotted are changed.
Figure 11.34: Interaction plot for factors temperature and pH, chemical reaction data.
From Figure 11.36, there is a stable region where >90% yield can
be obtained when Temperature and Stirring are kept high and pH is
kept below its mid-point. Since response surfaces are mainly used for
Optimisation designs, much emphasis will be placed on their
interpretation. When used in a Factor Influence Study, these plots
show the best direction to take for locating the optimal point.
Cube plots
Design stage
Following on from the chemical process design of Table 11.15, the
process engineer, being a clever person, has decided use DoE to
build the design and perform the analysis. After opening the Design-
Expert® package a number of design options were presented.
However, this process engineer has performed all of the necessary
homework and decides on using a design. The design table is
presented in Table 11.21.
Figure 11.37: Cube plot, chemical reaction data.
a 1 –1 –1 –1 69.5
b –1 1 –1 –1 48.6
c –1 –1 1 –1 66.8
abc 1 1 1 –1 64.1
d –1 –1 –1 1 44.1
abd 1 1 –1 1 99.5
acd 1 –1 1 1 83.2
bcd –1 1 1 1 68.6
Figure 11.38: Half-normal plot of effects for the design for chemical process data.
Frame 11.1: ANOVA table for reduced chemical process model with measure of curvature.
Frame 11.2: ANOVA table for reduced chemical process model with measure of curvature
removed.
Improve stage
The process engineer has been successful in finding a set of
conditions that will maximise the yield of a chemical process. The risk
of performing a fraction of the full design has paid off and a predictive
model has been established. The regression coefficients obtained
from the model can be used as a predictive model and the output
from Design-Expert® are shown in Frame 11.4.
The response surface for this model indicates that 95% yield is
possible when the following conditions are set for the process.
Temperature = 100°C
pH = 6
Stirring = 250 rpm
Pressure = 300 psi
By keeping the insignificant factor pressure at its low level, the
company can lower its energy costs and also reduce the scheduled
maintenance of the pump due to its lower workload. The process
engineer would now set these conditions up, run a few batches with
the improved settings to assess the batch to batch variability and
when satisfied with the results, would then monitor the future batch
yields using Statistical Process Control (SPC) tools and maybe
implement some Multivariate Statistical Process Control (MSPC) tools
to ensure the process set points for the input variables are always
maintained to produce maximum yield.
Figure 11.40: a) Residuals vs predicted and b) residuals vs run plots for chemical
process data.
Frame 11.4
There are three parts to the above equation, the linear part, the
quadratic part and the interaction part. After the data for an
optimisation design have been collected, it is the task of the analyst
to use the information from the ANOVA table to determine which
model form (if any) best fits the data in a similar way to the factorial
designs discussed previously in this chapter.
The analysis of an optimisation design will be presented by
example using data from a bread baking process. There is something
special about the smell and taste of freshly baked bread and if the
loaf is plump and soft, it also has visual appeal. A baker wishes to
optimise a process to simultaneously get the maximum volume of loaf
possible, but at the same time, retain a certain level of firmness. After
thinking about this problem, both objectives are diametrically
opposed to each other, i.e. if the volume of the loaf is too large, the
bread will contain a lot of air and therefore be softer, while a loaf of
low volume will be denser and possibly much harder. The steps taken
by the baker to optimise the loaf characteristics are described in the
following sections.
Optimisation—define stage
With subject matter knowledge, the baker was able to define which
factors would most likely influence loaf volume and firmness. These
are listed in Table 11.22 along with a short reasoning for their
selection.
The factors in Table 11.22 are all continuous and an overall centre
point is available for the design. Since it is possible to make many
loaves of bread in a single day, it was decided to run the experiment
in one block, using a CCD with the following objective for the two
response variables defined in Table 11.23.
Firmness is measured using a sensory scale, so it is prone to
more error compared to volume. With these objectives defined, an
appropriate design can be constructed.
Optimisation—design stage
Using Design-Expert®, a Response Surface model was selected and
the Central Composite Design was used. The three experimental
factors were setup as defined in Table 11.22 using three centre
points. This results in a total of 17 experimental runs. Two responses
were added to the design table.
The design table in standard order with all of the responses
entered after performing the experimental runs is provided in Figure
11.44.
The structure of the design table in standard order is described as
follows,
1) Experimental runs 1 to 8 define the 23 full factorial section of the
design.
2) Experimental runs 9–14 define the star points added to the factorial
design.
3) Experimental runs 15–17 define the centre points added to the
design.
Optimisation—analyse stage
The first step in the analysis stage is to determine the form of the
model to use, software such as Design-Expert® provides a summary
table for each response being analysed. The results of this summary
for the response Volume is provided in Figure 11.45. Since there are
two responses being analysed, the second response, Firmness will
be summarised after Volume has been analysed.
The software suggests a quadratic model is the best fit, based on
the p-value for the model being significant using the Whitcomb
Score, the lack of fit being just insignificant and this model has the
highest Adjusted and Predicted R-Squared values.
Figure 11.45: Model fit statistics for the response volume in the bread baking process.
Figure 11.46: Full ANOVA table for bread baking process fitting a quadratic model.
Figure 11.49: Predicted vs actual plot of bread volume, bread baking process data.
Optimisation—improve stage
When there is more than one response being measured, methods
such as numerical and graphical optimisation are required. For an
even more in-depth analysis of the joint optimisation procedure, the
method of Partial Least Squares Regression (PLSR, chapter 7) is also
suggested as it relates the responses directly to the input factors and
to each other.
In the case of the bread baking process, the optimal solution for
the joint optimisation of loaf volume and loaf firmness was found to
occur when the parameters in Table 11.24 were set for the process.
The baker would now run a number of trial batches at these
settings to establish consistency over batches, raw materials and
other bakers making the same bread as part of robustness testing.
Figure 11.52: Response surface of the response firmness, bread baking process.
Figure 11.53: Three-dimensional response surface of the response firmness, bread baking
process.
Figure 11.54: Overlay plot for joint optimisation of volume and firmness, bread baking
process.
Firmness 2.24
Table 11.25: Ranges of the process variables for the cooked meat
design.
The new design table, taking into account the constraints imposed
on the system is provided in Table 11.27.
Figure 11.55 b) shows that the “corners” of the factorial design
have been cut off for steaming and frying compared to the
experimental region represented by the full factorial design.
Depending on the number and complexity of the multi-linear
constraints to be taken into account, the shape of the experimental
region can be more or less complex. In the worst case, it may be
almost impossible to imagine! Therefore, building a design to screen
or optimise variables linked by multi-linear constraints requires
special methods of construction in order to be analysed using
methods such as MLR.
1 6 5 5
2 18 5 5
3 6 15 5
4 18 15 5
5 6 5 15
6 18 5 15
7 6 15 15
8 18 15 15
Figure 11.55: The cooked meat experimental region. a) Without and b) with constraints.
1 6 5 11
2 6 5 15
3 6 9 15
4 6 11 5
5 6 15 5
6 6 15 9
7 18 5 11
8 18 5 15
9 18 9 15
10 18 11 5
11 18 15 5
12 18 15 9
D-Optimal designs
Returning to the freezer meal example introduced earlier in this
section, where two constraints were placed on the variables steaming
time and frying time. Referring to Figure 11.55 b), the design has the
corners cut off the cube due to the two constraints, but how was this
design generated? In these cases, one of the alphabet optimal
designs are required to select the best possible subset of points to
include. The D-optimal principle, which consists in enclosing the
maximum volume into the selected points is described here, where it
is known that a selection of points will not exactly re-constitute the
entire experimental region of interest, but only the smallest possible
volume will be excluded.
When multi-linear constraints are introduced among the design
variables, it is no longer possible to build an orthogonal design. This
introduces a certain degree of dependency amongst the design
factors and as soon as the variations in one of the design variables
are linked to those of another design variable, orthogonality cannot
be achieved. These designs require a computer to generate them and
the analyst must have some idea of the number of points to include in
the design, up-front, based on an assessment of the shape that the
constraints impose on the design space.
The design is generated based on a grid of candidate points that
are spaced over the design region. Since the D-Optimal design aims
to maximise the volume that the candidate points are supposed to
enclose, mathematically, this is equivalent to minimising the
determinant of equation 11.18.
Figure 11.56: With only eight points, the enclosed volume is not optimal.
Mixture designs
0 ≤ xi ≤ 1 I = 1, 2, …p
Synergistic blending
Figure 11.61 shows how a theoretical response y might change as a
function of x1, x2 composition for a [3, 2] Simplex Lattice design.
The dotted straight line in Figure 11.61 is the theoretical linear fit
of the data based on there being no interaction between the
components x1 and x2 during blending. When the mixture
components blend in a synergistic manner, the components
positively enhance each other’s contribution to the response and thus
a non-linear situation occurs. There are now enough points in this
design to calculate a quadratic model.
Antagonistic blending
Antagonistic blending is the opposite of synergistic blending and its
effect on a theoretical response is shown in Figure 11.62.
Figure 11.60: Some simplex lattice designs for p = 3, linear, quadratic and cubic model
structures.
Cubic blending
There are certain situations where the nature of blending between two
or more components changes from synergistic to antagonistic as the
proportions change. This is an example of cubic (or even higher
polynomial) blending. Figure 11.63 shows the situation of cubic
blending for a theoretical response for the [3, 3] Simplex Lattice
design.
Figure 11.61: Synergistic blending in a [3, 2] simplex lattice design with a quadratic model.
Figure 11.64: Simplex centroid and augmented simplex centroid designs for p = 3
components.
The actual design space and the precision of estimates over the
design for this study are provided in Figure 11.65. The precision at
the centroid is lower than the edges of the design but this is to be
expected from a Simplex Lattice design, since it puts little emphasis
on analysing the interior of the design and more emphasis on the
edges of the design.
1 9 1 0 0 2.8 3.50
2 10 0 1 0 1.4 2.50
3 8 0 0 1 1.8 4.00
Frame 11.6
Figure 11.67: ANOVA table for the response cost, wine mixture design.
Figure 11.68: Normal plot of residuals for response preference, wine mixture design.
Figure 11.69: Response surface for response preference, wine mixture design.
Figure 11.70: Predicted vs actual plot for response preference, wine mixture design.
Figure 11.71: Response Surface plots for responses a) preference and b) cost, wine mixture
design.
Grenache 60
55 ≤ LSR ≤ 70
0 ≤ Aromatics ≤ 5
30 ≤ Isomerate ≤ 40
What does this design space look like? Starting with the p = 3
simplex, the limits imposed on LSR are first plotted in Figure 11.73.
The first thing to note is that the design space with the lower and
upper bounds on LSR imposed is no longer simplex shaped, i.e. the
design region cannot be modelled using the standard Shéffe models.
This is not a concern at the moment, because there are situations
when all constraints are imposed, the design space becomes a
simplex. The upper bound on Aromatics is now shown in Figure
11.74.
Again, the upper bound results in a non-simplex shape. Figure
11.75 shows the design space when the Isomerate bounds are
applied.
The final design space is trapezoidal in shape and thus cannot be
analysed using standard mixture polynomials. More on the
development and analysis of these designs will be provided in a later
section of this chapter. The following present some general rules
regarding the shape of the design space when lower and upper limits
are imposed on the mixture space.
Figure 11.73: Simplex design space for p = 3. LSR with a lower and an upper bound.
Figure 11.74: Simplex design space for p = 3. LSR with a lower and an upper bound,
aromatics with an upper bound.
60% ≤ Flour ≤ 100%
0% ≤ Egg ≤ 40%
0% ≤ Sugar ≤ 40%
Figure 11.75: Simplex design space for p = 3. All bounds applied to the components in the
gasoline design.
The upper bound on Egg remains the same at 40% and this
balances since the lower bound on Sugar is still 0% at this stage. The
Sugar upper bound is now implicitly changed to 30% for mass
balance if Flour is set at 60% and Egg at 10%. When the last lower
bound is placed on Sugar, this changes the entire set of upper
bounds, while all lower bounds remain intact. When Egg and Sugar
are both at their lower bounds, this accounts for 10% + 15% = 25%
of the total mixture, this implies that the upper bound on Flour is now
75% to account for their always being a minimum proportion of Egg
and Sugar in the mixture.
When there is 75% Flour in the mixture and Egg is at its lower
bound of 10%, there is currently a 15% difference between Sugar at
its lower bound, this is the exact lower bound to be imposed. Now, if
Flour is set to 60% and the lower bound on Sugar is 15%, the upper
bound on Egg is implicitly changed to 25%. The upper bound on
Sugar remains the same. The final constraints placed on the design
are provided as follows.
Figure 11.77: Pancake mixture design constrained by three lower bounds in real units.
Flour = 60 + 5 = 65%
Egg = 10 + 5 = 15%
Sugar = 15 + 5 = 20%
0% ≤ Flour ≤ 60%
0% ≤ Egg ≤ 100%
0% ≤ Sugar ≤ 100%
When the constraint for Egg is placed on the design, the following
further constraints are imposed,
0% ≤ Flour ≤ 60%
0% ≤ Egg ≤ 20%
20% ≤ Sugar ≤ 100%
When Flour is at 60% and Egg is at 20%, the lower bound of 20%
is imposed on Sugar. Now when the upper bound is imposed on
Sugar, the following constraints are calculated,
0% ≤ Flour ≤ 40%
0% ≤ Egg ≤ 20%
60% ≤ Sugar ≤ 100%
Figure 11.78: Mixture design space for p = 3 components for one, two and three upper
bounds imposed.
Define stage
A fruit punch is to be prepared by blending three types of fruit juice:
watermelon, pineapple and orange. The purpose of the manufacturer
is to use their large supplies of watermelons by introducing
watermelon juice, of little value by itself, into a blend of fruit juices.
Therefore, the fruit punch has to contain a substantial amount of
watermelon—at least 30% of the total. Pineapple and orange have
been selected as the other components of the mixture, since juices
from these fruits are easily obtainable and relatively inexpensive.
Figure 11.80: The U-simplex design space for the pancake mixture design.
Design stage
The manufacturer decides to use experimental design to find out
which combination of the three ingredients maximises consumer
acceptance of the taste of the beverage. The ranges of variation
selected and the upper and lower bounds for the experiment are
listed in Table 11.31.
Since it is the purpose of this experiment to optimise the fruit juice
blends, a design that covers the interior, including axial check blends
will be used. The design chosen was the Simplex Centroid design in
10 experimental runs. The design contains only one lower bound, this
means that the design space is simplex shaped and the usual Schéffe
polynomials can be applied. To generate this design, Design-Expert®
was used. The design table was sorted by Standard (Std) Order. The
experiments were run and the design table now includes all of the
responses for analysis (Figure 11.81).
The Real Contour plot of Figure 11.82 shows the actual design
space and the standard errors associated with the design. It also
shows the constrained design space with respect to the overall
unconstrained space.
Analyse stage
The Fit Summary suggests a Quadratic Model is the best fit for the
response Acceptance (Figure 11.83).
The ANOVA table for the quadratic model is provided in Figure
11.84.
It is noted in the ANOVA table that the model is significant. Only
the term Watermelon × Orange (AC) is insignificant. Note that in a
mixture model, main effects cannot be removed from the design due
to their dependencies on each other. This is why the Linear Mixture
effect is stated as the total main component contributions to the
model.
The term AC was removed and the ANOVA table for this modified
quadratic model is provided in Figure 11.85.
All terms are significant in the model and the residual is small
compared to the model sums of squares. This is reflected in the
supplementary statistics provided by the model (see Frame 11.7).
The Normal Probability plot of Residuals is shown in Figure 11.86.
Figure 11.84: ANOVA table for response, acceptance, fruit beverage design.
Figure 11.85: ANOVA table for response, acceptance, after the insignificant term AC was
removed, fruit beverage design.
Watermelon = 42%
Pineapple = 12%
Orange = 46%
Improve stage
The power of experimental design has been demonstrated again
using this case study. A constrained design, which forces there to be
a minimum amount (above zero) of a single component has shown
that an acceptable recipe for a fruit beverage is possible and can be
formulated from the three components used.
It is now the task of the company to make this recipe again (and
possibly use a smaller designed experiment around this target
formulation) to test the robustness of the recipe to small, but
deliberate changes. These would be used as a blind taste test
strategy on both a trained and untrained sensory panel to see if the
acceptance data is validatable. From there the company’s marketing
team can promote this new product to the general public.
Figure 11.86: Normal probability plot of residuals for response acceptance, fruit beverage
design.
Figure 11.87: Predicted vs actual plot for response acceptance, fruit beverage design.
Figure 11.88: Response surface plot of response acceptance, fruit beverage design.
Figure 11.89: Graphical optimisation of response acceptance, fruit beverage design.
The data in Figure 11.92 show a better fit of the linear model for
Friability and Hardness compared to Dissolution, however, there is
still a distinct curvature that requires a higher order model.
The form of the linear model for this analysis is,
Figure 11.92: Predicted vs reference plots for friability and hardness for tablet data.
Figure 11.93: PLSR overview of the interaction and squares model for the tablet data.
11.4 References
[1] Box, G.E.P., Hunter, W.G. and Hunter, J.S. (1978). Statistics for
Experimenters, An Introduction to Design, Data Analysis and
Model Building. John Wiley & Sons.
[2] Montgomery, D.C. (2005). Design and Analysis of Experiments,
6th Edn. John Wiley & Sons.
[3] Anderson, M.J. and Whitcomb, P.J. (2015). DOE Simplified:
Practical Tools for Effective Experimentation, 3rd Edition. CRC
Press. https://doi.org/10.1201/b18479
[4] Myers, R.H. and Montgomery, D.C. (2002). Response Surface
Methodology, Process and Product Optimization Using
Designed Experiments, 2nd Edn. John Wiley & Sons.
[5] Cornell, J. (2002). Experiments with Mixtures, Designs, Models
and the Analysis of Mixture Data, 3rd Edn. John Wiley & Sons.
https://doi.org/10.1002/9781118204221
[6] Smith, W.F. (2005). Experimental Design for Formulation.
American Statistical Association and the Society for Industrial
and Applied Mathematics.
https://doi.org/10.1137/1.9780898718393
[7] Crosier, R.B. (1984). “Mixture experiments: geometry and
pseudo-components”, Technometrics 28, 209–216.
https://doi.org/10.1080/00401706.1984.10487957
Chapter 12. Factor rotation and
multivariate curve resolution—
introduction to multivariate data
analysis, tier II
This chapter will introduce to the reader selected topics from tier II of
multivariate data analysis; issues and methods that definitely cannot
be understood, far less mastered, without significant experience with
the introductory curriculum in all preceding chapters, especially the
demonstration data analyses and comprehensive hands-on exercise
accomplishments. This chapter can be viewed as a bridge between
the current ideas in this book and more advanced multivariate data
analysis methods, which the authors strongly suggest that the reader
pursue after mastering the basics.
The line rotated loadings plot shows that the original correlation
between the PCs around 944 nm is now minimised and this indicates
that in the original PCA, fat and moisture are correlated. However,
during rotation, these two factors now separate the two effects,
revealing that a simple structure of fat and moisture content is
present in the data set. This example will be described in more detail
in section 12.6, where the method of multivariate curve resolution
(MCR) will be applied to the data.
Overall, rotation methods can lead to simple structure, which in
some cases is demonstrably easier to interpret than the original PCA
space. The rotated loadings may still not have real physical meaning,
however, but in the case of the fat in fish example, the loadings meet
the Varimax criteria of a small number of large loadings and a large
number of near zero loadings for each factor. These rotated factors
have better physical meaning in the sense that fat and moisture are
typically correlated in samples of biological nature.
The next section describes the effect of PC rotation under certain
predefined constraints known as multivariate curve resolution (MCR).
Physico–chemical constraints
One of the most recent progresses in chemical constraints refers to
the implementation of a physico–chemical model into the multivariate
curve resolution domain. In this approach, the concentration profiles
of compounds involved in a kinetic or a thermodynamic process are
shaped according to a relevant chemical law. Such a strategy has
been used to reconcile the separate worlds of hard- and soft-
modelling and has enabled the mathematical resolution of chemical
systems that could not be successfully tackled by either of the two
pure methodologies alone. The strictness of a hard model constraint
dramatically decreases the ambiguity of the constrained profiles and
provides fitted parameters of physico–chemical and analytical
interest, such as equilibrium constants, kinetic rate constants and
total analyte concentrations. The soft part of the algorithm allows for
modelling of complex systems, where the central reaction system
evolves in the presence, and as a function of absorbing interferences.
Refer to the literature on kinetic modelling and other related methods,
particularly the work of Maeder [13].
Returning to PCA of the chemical reaction data, MCR is now
applied to the data set using the constraints of non-negativity on both
concentration profiles and spectra as well as the constraint of uni--
modality based on subject-matter expertise regarding the underlying
reaction kinetics (as always, the more domain-specific knowledge
available, the better and the more relevant data analysis possible).
This constrained MCR overview is provided in Figure 12.12.
First inspection of the constrained MCR indicates that the three-
component model now results in significantly minimised sample
residuals. The component concentrations plot shows that for
component 1, there is a monotonic decrease (uni-modality) in the
concentration of this component as time progresses. This is
indicative of a starting material being consumed by the reaction. The
blue curve in the component spectra plot provides the estimated
spectra for the starting material. As the reaction progresses, the
appearance of component 3 reaches a unimodal point where no more
product is being generated and its spectrum is defined by the green
curve in the component spectrum plot.
It is the generation of the second component that establishes
MCR as the more relevant data analysis approach for these data vs
PCA. In the PCA overview, Figure 12.10, only two sources of
variability were observed—these can be understood as PCA
attempting to find the greatest sources of variability, in this case, the
consumption of the reactant and the generation of the product,
precisely the design feature of PCA as variance maximisation. In
contrast, a suitably constrained MCR was not only able to describe
the consumption and formation kinetics, but was also able to reveal
the transition state spectrum (the red curve) reflecting a transition
state compound in the reaction. In a sense MCR “discovered” this
third component, which was hidden or swamped by the
overpowering other two component variances.
The appropriately constrained MCR now provides the relevant
basis for detailed mechanistic assessment of the reaction profile and
provides a spectral assessment of transition state that may not have
been able to be separated by any other means, both physical or
mathematical.*
Figure 12.12: MCR overview of the chemical reaction data using UV/vis spectroscopy.
After developing a first MCR and from a review of PCA from the initial
data set, it is important that the analyst checks the estimated number
of pure components and attempts an initial interpretation of the
profiles of those components. Software programs typically provide an
option to adjust the analysis to be more or less sensitive to the
component estimates added to the analysis. There are a number of
situations that can occur that will require tuning of the sensitivity of
those components such that they do not result in meaningless
results. Some cases are listed as follows.
Case 1: The estimated number of pure components is larger than
expected. In this situation, the best action is to reduce the sensitivity
of the analysis, using software, such that a more meaningful result is
obtained.
Case 2: When there are no prior expectations about the number of
pure components, but some of the extracted profiles look very noisy
and/or two of the estimated spectra are very similar, this indicates
that the actual number of components is probably smaller than the
that estimated. In this case, the best action again is to reduce the
sensitivity.
Case 3: When it is known that there are at least n different
components with varying concentrations in the system, and the
estimated number of pure components is smaller than n, this is a
case where the best action is to increase sensitivity to pure
components such that the analysis results in the expected number of
components.
Cases 1–3 above require the use of software to adjust sensitivities
and this is a feature of The Unscrambler® in the MCR functionality.
The default value is 100% sensitivity and this value can be adjusted
either side of this default depending on the situation encountered
with the data being modelled.
12.8.1 Residuals
The residuals are a measure of the fit (or rather, lack-of-fit) of the
model. The smaller the residuals, the better the fit. MCR residuals can
be studied from three different points of view.
Variable residuals: these are a measure of the variation remaining
in each variable after k components have been estimated.
Sample residuals: these are a measure of the distance between
each sample and its model approximation.
Total residuals: these are a measure of how much variation in the
data remains to be explained after k components have been
estimated. Their role in the interpretation of MCR results is similar
to that of variances in PCA.
As is the case for all multivariate analysis, the available data may be
more or less “clean” or “precise” when building the first curve
resolution model. The main tool for diagnosing outliers in MCR
consists of using the plot of sample residuals. Any sample that stands
out on the plots of sample residuals (either with MCR fitting or PCA
fitting) is a possible outlier.
To find out more about such a sample (Why is it outlying? Is it an
influential sample? Is that sample disturbing the model?), it is
recommended to run a PCA on the data. If an outlier is justifiably
removed, recalculation of the MCR model without that sample may
lead to a more interpretable analysis. In MCR, some of the available
variables—even if they are no more “noisy” than the others—may
also contribute poorly to the resolution, or even disturb the results.
The two main cases are:
Non-targeted wavelength regions: these variables carry virtually
no information that can be of use to the model;
Highly overlapped wavelength regions: several of the estimated
components have simultaneous peaks in those regions, so that
their respective contributions are difficult to disentangle (difficult to
unscramble).
The main tool for diagnosing noisy variables in MCR consists of
the plots of variable residuals. Any variable that stands out on such a
plot (either with MCR fitting or PCA fitting) may be disturbing the
model, thus reducing the quality of the resolution; try recalculating the
MCR model without such suspect variables.
Figure 12.15: EMSC-corrected spectra of fat in fish using estimated fat spectrum from
MCR.
Figure 12.16: PLSR model of fat in fish using EMSC/MCR preprocessing.
Figure 12.17: Updated PLSR model of fat in fish after the removal of justified outliers.
Figure 12.18: a) MSC preprocessed and b) EMSC preprocessed fat in fish calibration
models.
12.11 References
[1] Jackson, J.E. (1991). A User’s Guide to Principal Components.
John Wiley & Sons. https://doi.org/10.1002/0471725331
[2] Harman, H.H. (1976). Modern Factor Analysis, 3rd Edn.
University of Chicago Press, Chicago.
[3] Westad, F. and Kermit, M. (2009). Independent Component
Analysis in Comprehensive Chemometrics. Elsevier Science, Ch.
2.14, pp. 227–248.
[4] Kaiser, H.F. (1958). “The varimax criterion for analytic rotation in
factor analysis”, Psychometrika 23, 187–200.
https://doi.org/10.1007/BF02289233
[5] Mardia, K.V., Kent, J.T. and Bibby, J.M. (1979). Multivariate
Analysis. Academic Press, London.
[6] Neuhaus, J.O. and Wrigley, C. (1954). “The quartimax method:
an analytic approach to orthogonal simple structure”, Brit. J.
Stat. Psych. 7(2), 81–91. https://doi.org/10.1111/j.2044-
8317.1954.tb00147.x
[7] Darton, R.A. (1980). “Rotation in factor analysis”, The
Statistician 29, 167–194. https://doi.org/10.2307/2988040
[8] Malinowski, E.R. and Howery, D.G. (1980). Factor Analysis in
Chemistry. John Wiley & Sons.
[9] Saunders, D.R. (1953). “An analytic method for rotation to
orthogonal simple structure”, Princeton, Educational Testing
Service Research Bulletin 53–10.
[10] Crawford, C.B. and Ferguson, G.A. (1970). “A general rotation
criterion and its use in orthogonal rotation”, Psychometrika
35(3), 321–332. https://doi.org/10.1007/BF02310792
[11] Browne, M.W. (2001) “An overview of analytic rotation in
exploratory factor analysis”, Multivariate Behavioural Research,
36(1), 111–150.
[12] Tauler, R. and de Juan, A. (2006). “Multivariate curve
resolution”, in Practical Guide to Chemometrics, 2nd Edition.
Taylor & Francis, Ch. 11.
https://doi.org/10.1201/9781420018301.ch11
[13] Maeder, M. and Neuhold, Y.M. (2006). “Kinetic modeling and
multivariate measurements with non-linear regression”, in
Practical Guide to Chemometrics, 2nd Edition. Taylor & Francis,
Ch. 7.
[14] Rutan, S.C., de Juan, A. and Tauler, R. (2009). “Introduction to
multivariate curve resolution”, in Comprehensive Chemometrics,
Vol. 2. Elsevier, Ch. 2.15, pp. 227–248.
[15] Maeder, M. and Zilian, A. (1988). “Evolving factor analysis, a
new multivariate technique in chromatography”, Chemometr.
Intell. Lab. Syst. 3, 205–213. https://doi.org/10.1016/0169-
7439(88)80051-0
[16] Multivariate curve resolution web page,
http://www.cid.csic.es/homes/rtaqam/ [accessed 11 January
2017].
[17] Tauler, R. (1995). “Multivariate curve resolution applied to
second order data”, Chemometr. Intell. Lab. Syst. 30, 133–146.
https://doi.org/10.1016/0169-7439(95)00047-X
[18] de Juan, A. and Tauler, R., “Chemometrics applied to unravel
multicomponent processes and mixtures. Revisiting latest
trends in multivariate resolution”, Anal. Chim. Acta 500, 195–
210 (2003). https://doi.org/10.1016/S0003-2670(03)00724-4
* PCA, and MCR even more so, have been likened to “mathematical data chromatography”.
In this analogy PCA is going about its work using a standard column (length), fixed
temperature setup while MCR is using flexible column length and advanced temperature
programming etc.
Chapter 13. Process analytical
technology (PAT) and its role in the
quality by design (QbD) initiative
13.1 Introduction
The pharmaceutical and related industries have been given incentive
to adopt state-of-the-art process monitoring and control strategies,
much like other industries have been doing for many years.
Traditional arguments coming from the industry, such as “the
pharmaceutical industry is different to other industries”, have
sometimes stifled the opportunity to become innovative, however,
this situation is gradually changing. Is the pharmaceutical industry
different from other industries? “Yes, absolutely”; it deals with the
treatment of sick people and the need for high quality products that
do their job is imperative. Is the pharmaceutical industry different
from other industries from a manufacturing perspective? “Absolutely
not”—all industries share the same issues regarding product quality
and process efficiency. Within the industry, contrary arguments that
try to maintain the status quo, may be blocking the opportunity to
improve, where typically it is cited that a regulatory agency “will not
accept” substantial changes to the process or product, sometimes
without even “testing the water”.
Indeed, it used to be a very expensive process to change the
market dossier of a product if any process or product changes were
to be made post approval, and this increased the resistance to make
any meaningful process changes, even though they were necessary.
In the early 2000s the US FDA, under the guidance of Dr Ajaz
Hussain, analysed all of the warning letters issued to companies
based on process deviations and instituted what is known today as
the Code of Good Manufacturing Practices for the 21st Century
(cGMPs for the 21st Century [1]). The concept and final paper was
called the “Scientific, Risk-Based Approach” to pharmaceutical
manufacturing. Dr Hussain highlighted the notable lack of innovation
in pharmaceutical manufacturing at a conference in Singapore in
2007, where he stated that the manufacturers of M&Ms have much
tighter controls over their coating process than the pharmaceutical
industry has on tablet coating processes. Should this not raise
concern? Is not the pharmaceutical industry held up as the gold
standard of product quality and manufacturing excellence in the
public eye?
Hussain’s main focus was to encourage industry to adopt a
paradigm shift from an 18th century approach to quality to a 21st
century approach. The area of precision agriculture has utilised state-
of-the-art technology for many years for fertilisation management and
irrigation planning for crops using near infrared (NIR) spectroscopy
and chemometrics, methods only recently adopted to any great
degree by the pharmaceutical industry. cGMPs for the 21st Century
was a concerted attempt to help industry realise that innovation does
not stop after the R&D stage and should continue throughout the
entire product’s lifecycle. In order to make innovation work, a new
mindset is required throughout an entire organisation, where a “can
do’ attitude is adopted, rather than a reactive and pre-emptive one.
The reality is that “quality costs”, and cannot be considered a red
line on an accountants’ ledger book. Paying for quality upfront will
naturally lead to cost-effective manufacture of the highest standard of
product. To aid industry in the implementation of better quality
systems, the Quality by Design (QbD) initiative was established such
that scientists and engineers could implement the latest advances in
process monitoring and control systems with a regulatory framework
to support such implementation. QbD therefore requires advanced
process sensor technology and modern quality systems to enable
such implementations. This is where the general premise of Process
Analytical Technology (PAT) comes to the fore.
This chapter aims to provide both new and existing practitioners
with an overview of the QbD and PAT initiatives, their interrelationship
and how this all ties into the key analysis methods of DoE,
chemometrics and TOS. The most important guidance documents
will be reviewed in a pragmatic manner along with a practical,
implementation approach to the pharmaceutical quality system
(PQS). QbD is the embodiment of all of the concepts discussed in this
textbook so far—from sampling, to appropriate technology, to the
design of rational experiments right through to better process
understanding and finally a system for monitoring and controlling
processes to meet the highest possible levels of validity.
NIR spectroscopy has been the major PAT used until recently and
has enjoyed the status of being the preferred technology for use in
the pharmaceutical industry, mainly due to its versatility and non-
destructive sampling. In more recent times, there has been an
emergence of other spectroscopic and non-spectroscopic tools that
have found their way into the PAT practitioners’ toolkit including
Raman spectroscopy, terahertz spectroscopy, improvements in mid-
infrared spectroscopy, particle size analysis (PSA) and many more
tools (particularly based around imaging) are becoming available all
the time.
Case study, NIR for fluid bed drying monitoring and control
(real-world PAT implementation)
In one implementation, a FT instrument was coupled to two fluid bed
driers (FBD) using a multiplexer in a manufacturing facility of generic
products for one of its product formulations. The manufacturer was
experiencing a major bottleneck and downstream processing issues
for this product and isolated the FBD as the root cause of the
problem.
Careful analysis of the data being generated by QA suggested
that the loss on drying (LOD) of the product was not only missing
target more times than not, but also the uniformity of moisture in the
powder bed was also non-uniform. This issue suggested that the
initial three-batch validation approach was not robust enough to pick
up this process flaw! The secret to this processing issue was found
through the use of NIR. After a strict design of experiments procedure
was carried out to optimise the position of the fibre optic probe in the
drier, initial trials on real production batches were conducted. The
method of PCA targeted at the 1930 nm (moisture) region of the
spectrum was observed in scores and loadings space. This showed
that the product was dry in 10 min (compared to the validated 40 min
specification). It was interesting to note that the SOP for the process
stated that after 10 min, the process should be stopped, the granules
should be remixed and the bed placed back into the FBD for
continued drying. This raised concerns as the need for extra manual
handling could introduce potential contamination.
After a review of the data generated by NIR, when the bed was
spatially sampled (unfortunately grab sampled in this case, chapter
3), the data revealed that the side walls, where the NIR probe was
located were typically much drier than the inner sections of the bed.
This indicated a lack of fluidisation in the process and this was why
there was a need for manual remixing of the granules. The next step
was to perform an engineering study on the process to better
understand why the bed was not fluidising in the first 10 min.
In a modern FBD, the air vents to the bed are usually vertically
oriented with respect to the FBD column, providing the most efficient
airflow to the wet mass. Also, these modern FBD systems have a
dehumidification system that controls the moisture level of the heated
air that dries the powder. In this particular case, the FBD was an older
system with a side air vent and no dehumidification system. These
were the major causes of the lack of robustness in the process and
an engineering solution was required to initiate fluidisation without
manual intervention. This was provided in the form of a feature of the
drier known as the “product loosening” button that automatically
induced a fluidised bed, however, it was not used in the past as there
was no way of triggering when to apply the button until NIR came
along.
To assess if this functionality was the key to the problem, NIR was
used to monitor when the bed reached its “dry” state and then the
button was manually pressed. The function of the product loosening
button is to create an instantaneous pressure and release cycle that
redistributes the powder without manual intervention. The
observation was that the moisture monitored by NIR rose sharply and
clear fluidisation was visible in the sight glass of the drier. The powder
bed then dropped to near dry around the 20 min point where the
product loosening button was used again. Only a small increase in
moisture was observed at this time point and the product reached a
stable endpoint after only 25 min. Although the reduction of the
drying time by 15 mins compared to the validated process did not
seem to be a great gain in time saving, it represented a situation of
greater efficiency where a product of desired state was attained.
However, the real benefits of the NIR method are as follows,
1) The FBD can be operated in a more efficient manner without
complete system reengineering.
2) NIR provided a key insight into the process mechanism and
allowed the engineers to understand the root cause of the problem.
3) Although only 15 min was gained on the process efficiency, this
does not also take into account that the bed required manual
remixing twice (10 min per remix) and if the product did not meet
LOD specification after 40 min, it had to be returned to the FBD,
dried for a further 5 min and an LOD taken again. The LOD test
required 10 min and if it had to be performed twice with re-drying,
a total time of 30 min was added to the process. NIR therefore
allowed a reduction of over an hour per mix compared to the
current state and with 8 mixes per batch, the math speaks for itself.
4) NIR allowed the operators to monitor the product to its desired
state without any manual intervention. This resulted in less process
issues downstream compared to the current implementation.
5) Greater quality was built into the process by design.
It was common occurrence with the product that the re-drying
step was required for each mix and therefore 8 h (or the equivalent of
one working shift) was required just to account for a poorly validated
process. The NIR was implemented as part of a control system that
automatically implemented the product loosening feature at the
appropriate time, thus allowing improved granule formation, improved
material flows, produced less issues downstream and as an added
benefit to the organisation, allowed four extra batches to be
produced per month without the need for factory expansion.
The following represents a typical cost justification for
implementing PAT in a QbD environment.
Initial system cost and development time (including salaries): 300 K
USD
Operator costs and energy consumption estimate (per hour): 500
USD
Typical cost on eight mixes per batch current system: 5 K USD
Cost to manufacture four batches per month (less materials): 16
K USD
Operating cost based on eight mixes per batch using NIR: 1.5 K
USD
Market value of batch (internal): 200 K USD
Revenue increase through four extra batches: 800 K USD
Increased production less initial equipment outlay: +500 K USD
Payback period: 1 month
These figures are based on costs and available instrumentation at
the time of this development, however, with miniaturisation of
instrumentation (and subsequently lower costs), the figures stated
above are achievable and realistic for this type of implementation.
For a more complete description of the NIR method and its
applications, the reader is referred to the excellent handbook by
Burns and Cuirczak [22] and the concise reviews by Swarbrick [16,
23]
1) Preprocess, centre and scale the new data before the batch model
can be applied.
2) Estimate the new scores tnew,a = xnewpa for the a components used
to develop and validate the batch model.
3) Project these scores onto the trajectory for estimation of the
relative time, distance to the trajectory and distance to the model.
Figure 13.10 shows the projection of a new batch onto the
developed batch model and the details of the projection are
discussed in the text that follows.
The main observations are made as follows,
1) The new batch started prior to the common starting point of the
batch model. This indicates that the conditions were immature
compared to the common situation, therefore the batch model
knows, through projection, that it has to wait until a point projects
into the design space of the batch model before monitoring and
control begins.
2) There were a few points which transgressed outside the limits of
the design space (these were deliberately set), however, through
use of APC, the batch can be corrected before any major quality
issues occur.
3) Note that the spacing between the points is not even. This is the
major advantage of RTM over other batch modelling approaches, if
the reaction slows down, stalls or even reverses (as shown at score
coordinates (–1,0), as long as the batch remains within the derived
common limits, there is no reason to suggest that the batch is
deviating.
Figure 13.10: Projection of a new batch onto the chemical reaction process batch
model.
Figure 13.11: Projection of temperature probe-1 values into relative time space for
chemical reaction process.
Material 1 1 0 0 0 0
Material 2 0 1 0 0 0
Material 3 0 0 1 0 0
Material 4 0 0 0 1 1
Material 5 0 0 0 1 1
Figure 13.14: Generic flowchart of the hierarchy defined for the classification–
classification case for the five raw material library.
Figure 13.21: Data fusion for the case of a complex process system.
Figure 13.24: Variogram expression of a situation in which the total process measurement
system gets progressively worse and is no longer able to depict the true, underlying
process variation (sill). QC of the measurement system, RSV1 – dim is 20, 60, 90% of the
sill, respectively. The vertical bar represents the true process variation, progressively
more-and-more swamped by measurement system failure. This system check is obtained
by variographic analysis of the process data themselves, i.e. it can be carried out in an
on-line, in real-time modus.
Idea to feasibility
There are many opportunities to implement PAT into a process, but
are all valid? The answer is no. The PAT implemented must be fit for
purpose and this is where a small, but comprehensive feasibility trial
is required. It has been the authors’ experience over many years that
too many companies run endless R&D trials to gain confidence in the
technology. When these trials exceed two to three weeks, it can be
safely assumed that this will never reach a real process and is just a
“tyre kicking” exercise. As with any analytical method development,
the production of a small, but concise protocol to accept/reject the
technology is required that has the buy in of all key stakeholders.
Modifications to the protocol are allowed, based on unanticipated
findings, however, this feasibility trial must be performed with the end
intent of implementing the technology and not to keep it going forever
in a repetitive loop.
13.9 References
[1] US FDA (2004). Pharmaceutical cGMPs for the 21st Century – A
Risk Based Approach, Final Report.
http://www.fda.gov/drugs/developmentapprovalprocess/manufacturing/que
[accessed 3 June 2015].
[2] Esbensen, K.H., Roman-Ospino, A.D., Sanchez, A. and
Romanach, R.J. (2016). “Adequacy and verifiability of
pharmaceutical mixtures and dose units by variographic
analysis (Theory of Sampling) – A call for a regulatory paradigm
shift”, Int. J. Pharm. 499, 156–174.
https://doi.org/10.1016/j.ijpharm.2015.12.038
[3] Code of Federal Regulations 21 CFR Part 211, Current Good
Manufacturing Practice for Finished Pharmaceuticals, Food and
Drug Administration. Department of Health and Human
Services.
[4] ICH Harmonized Tripartate Guideline Q2(R1) (1997). “Validation
of analytical procedures: text and methodology”, Federal
Register 62(96), 27463–27467.
[5] ICH Harmonized Tripartate Guideline Q8(R2) (2009).
“Pharmaceutical development”, Federal Register 71(98).
[6] Dickens, J.E., (2010). “Overview of process analysis and PAT”,
in Process Analytical Technology, Ed by Bakeev, K. John Wiley
& Sons, NY, pp. 1–14.
https://doi.org/10.1002/9780470689592.ch1
[7] ICH Harmonized Tripartate Guideline Q9 (2009).
“Pharmaceutical risk management”, Federal Register 71(106).
[8] ICH Harmonized Tripartate Guideline Q10
(2009).“Pharmaceutical quality system”, Federal Register 74(66).
[9] GAMP®5: A Risk-Based Approach to Compliant GxP
Computerized Systems.
http://www.ispe.org/publications/gamp4togamp5.pdf [accessed
6 January 2017].
[10] ICH Harmonized Tripartate Guideline Q11 (2009). “Development
and manufacture of drug substances (chemical entities and
biotechnological/biological entities)”.
[11] US FDA, Guidance for Industry – Process Validation: General
Principles and Practices.
http://www.fda.gov/downloads/Drugs/Guidances/UCM070336.pdf
[accessed 6 January 2017].
[12] US FDA (2004). Guidance for Industry: PAT – a framework for
innovative pharmaceutical manufacturing and quality assurance.
http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinform
[13] Esbensen, K.H. and Paasch-Mortensen, P. (2010). “Process
sampling: theory of sampling, the missing link in process
analytical technologies (PAT)”, in Process Analytical
Technology, Ed by Bakeev, K. John Wiley & Sons, NY, pp. 37–
80. https://doi.org/10.1002/9780470689592.ch3
[14] Bakeev, K. (Ed.) (2010). Process Analytical Technology, 2nd
Edn. John Wiley & Sons, NY.
[15] Plugge, W. and van der Vlies, C. (1992). “The use of near
infrared spectroscopy in the quality control laboratory of the
pharmaceutical industry”, J. Pharm. Biomed. Anal. 10(10–12),
797–803. https://doi.org/10.1016/0731-7085(91)80083-L
[16] Swarbrick, B. (2016). “Near infrared (NIR) spectroscopy and its
role in scientific and engineering applications”, in Handbook of
Measurement in Science and Engineering, Vol. 3, Ed by Kutz, M.
John Wiley & Sons, pp. 2583–2608.
https://doi.org/10.1002/9781119244752.ch71
[17] Berntsson, O., Danielsson, L.-G. and Folestad, S. (2001).
“Characterization of diffuse reflectance fiber probe sampling on
moving solids using a Fourier transform near-infrared
spectrometer”, Anal. Chim. Acta 431(1), 125.
https://doi.org/10.1016/S0003-2670(00)01313-1
[18] Blanco, M., Gozalez Bano, R. and Bertran, E. (2002).
“Monitoring powder blending in pharmaceutical processes by
use of near infrared spectroscopy”, Talanta 56, 203–212.
https://doi.org/10.1016/S0039-9140(01)00559-8
[19] Besseling, R., Damen, M., Tran, T., Nguyen, T., van den Dries,
K., Oostra, W. and Gerich, A. (2015). “An efficient, maintenance
free and approved method for spectroscopic control and
monitoring of blend uniformity: The moving F-test”, J. Pharm.
Biomed. Anal. 114, 471–481.
https://doi.org/10.1016/j.jpba.2015.06.019
[20] Muzzio, F.J., Alexander, A., Goodridge, C., Shen, E., Shinbrot,
T., Manjunath, K., Dhodapkar, S. and Jacob, K. (2004). “Solids
mixing”, in Handbook of Industrial Mixing: Science and Practice,
Ed by Paul, E.L., Atiemo-Obeng, V.A. and Kresta, S.M. John
Wiley & Sons.
[21] Muzzio, F.J., Robinson, P., Wightman, C. and Brone, D. (1997).
“Sampling practices in powder blending”, Int. J. Pharm. 155,
153–178. https://doi.org/10.1016/S0378-5173(97)04865-5
[22] Burns, D.A. and Ciurczak, E.W. (2008). Handbook of Near-
Infrared Analysis, 3rd Edn. CRC Press.
[23] Swarbrick, B. (2014). “Advances in instrumental technology,
industry guidance and data management systems enabling the
widespread use of near infrared spectroscopy in the
pharmaceutical/biopharmaceutical sector”, J. Near Infrared
Spectrosc. 22, 157. https://doi.org/10.1255/jnirs.1121
[24] Lewis, I.R. and Edwards, H.G.M. (2001). Handbook of Raman
Spectroscopy, From the Research Laboratory to the Process
Line. CRC Press.
[25] http://www.kosi.com/raman-spectroscopy/phat-probe-
head.php [accessed 16 June 2016].
[26] Coates, J.P. (2010). “Infrared spectroscopy for process
analytical applications”, in Process Analytical Technology, Ed by
Bakeev, K. John Wiley & Sons, NY, pp. 157–193.
https://doi.org/10.1002/9780470689592.ch6
[27] Westad, F., Gidskehaug, L., Swarbrick, B. and Flåten, G.-R.
(2015). “Assumption free modelling and monitoring of batch
processes”, Chemometr. Intell. Lab. Syst. 149(B), 66–72.
https://doi.org/10.1016/j.chemolab.2015.08.022
[28] Jørgensen, P., Pedersen, J., Jensen, E.P. and Esbensen, K.H.
(2004). “On-line batch fermentation process monitoring –
introducing biological process time”, J. Chemometr. 18, 81–91.
https://doi.org/10.1002/cem.850
[29] Bro, R. (1997). “PARAFAC. Tutorial and applications”,
Chemometr. Intell. Lab. Syst. 38, 149–171.
https://doi.org/10.1016/S0169-7439(97)00032-4
[30] Smilde, A., Bro, R. and Geladi, P. (2004). Multiway Analysis,
Applications in the Chemical Sciences. John Wiley & Sons.
https://doi.org/10.1002/0470012110
[31] Nomikos, P. and MacGregor, J.F. (1994). “Monitoring batch
processes using multiway principal component analysis”,
AICHE J. 40, 1361–1375.
https://doi.org/10.1002/aic.690400809
[32] Kassidas, A., MacGregor, J. and Taylor, P. (1998).
“Synchronization of batch trajectories using dynamic time
warping”, AICHE J. 44, 864–875.
https://doi.org/10.1002/aic.690440412
[33] Wold, S., Kettanch, N., Friden, H. and Holmberg, A. (1998).
“Modelling and diagnostics of batch processes and analogous
kinetic experiments”, Chemometr. Intell. Lab. Syst. 44, 331–340.
https://doi.org/10.1016/S0169-7439(98)00162-2
[34] Swarbrick, B. (2016). “Chemometrics for near infrared
spectroscopy”, NIR news 27(1), 39–40.
https://doi.org/10.1255/nirn.1584
[35] Brennan, Z., FDA Allows First Switch From Batch to Continuous
Manufacturing for HIV Drug. http://raps.org/Regulatory-
Focus/News/2016/04/12/24739/FDA-Allows-First-Switch-From-
Batch-to-Continuous-Manufacturing-for-HIV-
Drug/#sthash.0eGkzDLr.dpuf [accessed 18 June 2016].
[36] George, M.L., Rowlands, D., Price, M. and Maxey, J. (2005). The
Lean Six Sigma Pocket Toolbook. McGraw Hill.
[37] Yang, K. and El-Haik, B. (2003). Design for Six Sigma, A
Roadmap for Product Design, McGraw Hill.
[38] Box, G.E.P., Hunter, W.G. and Hunter, J.S. (1978). Statistics for
Experimenters, An Introduction to Design, Data Analysis and
Model Building. John Wiley & Sons.
[39] Swarbrick, B., Grout, B. and Noss, J. (2005). “The rapid, at-line
determination of starch content in sucrose-starch blends using
near-infrared reflectance spectroscopy; a process analytical
technology initiative”, J. Near Infrared Spectrosc. 13, 1–8.
https://doi.org/10.1255/jnirs.451
[40] Moore, R. (2013). Making Common Sense Common Practice.
Reliability Web Publishing.
Index
A
Abstract variables 391
Accuracy 59, 216
Additive effects 123, 124, 129
Advanced process control (APC) 429
Alternative hypothesis 26
Ambiguity resolution 438
Analysis level 431
Analysis of variance (ANOVA) 18, 319
Analyte 66
At-line 421
Autocorrelation 50
Auto-scaling 107, 108, 129
Averaging 109
B
Baseline shift 122
Batch trajectory 435
Bias 15, 197
Big data solutions 431
Bilinear methods 10
Blind source separation 399
Block effect 313
Box–Behnken. See Design of experiments (DOE)
Box plot 24
C
Calibration set 163
Proper training data 235
Requirements 163
Training set 163
Training stage 265
Categorical variables 309
Category variables. See Variables
Central limit theorem 33, 225
Centring 77, 93
Chromatography 114
Classification 9, 263
Basic steps 266
Classification stage 265
Distance metric 257
Modelling power 276
Pattern cognition 255
Pattern recognition 267
Si vs Hi 270
Supervised classification 255
Unsupervised classification 255
Classification table 266
Closed modus 277
Cluster analysis 255
Hierarchical cluster analysis 258
Hierarchical cluster analysis (HCA)
Agglomerative methods 258
Dendrogram 259
Divisive methods 258
k-Means clustering 256
Nearest neighbour distance 256
Ward’s method 259
Coefficient of variation 246, 328
Collinearity 169, 281
Composite sample 66
Confusion matrix 289
Continuous improvement 419
Continuous manufacturing systems (CMS) 419, 429, 442
Control charts 45
Control level 430
Control limits 47
Corrective and preventative maintenance (CAPA) 419
Correct sampling 246
Correlation 4, 48, 73, 142
Covariance 4, 48, 142
Critical process parameters (CPP) 418
Critical quality attributes (CQA) 418
Cross-validation. See Validation
Cross-validation vs test set validation 237
D
Data
Description 9
Fusion 445
Matrix 69
Dimension 70
Representativity 223
Structure 7, 93
Decomposition 79, 93
Deflation 178
Degrees of freedom (DF) 17, 36, 37, 320, 324
Derivation 122
Descriptive statistics 25, 139, 256
Design for Six Sigma 13, 444
Design of experiments (DOE) 233
Blocking 313, 329
Centre samples 309
Cube plots 343
Cubic blending 364
Curvature 330
D-Optimal designs 358
Efects
Main effect 302
Effects
Definition and calculation 303
Hierarchy of effects 322
Interaction 302
Interaction effect 306
Main 302
Main effect 306
Factor influence designs 311
Full factorial designs 301
Full cubic 363
Mixture designs 360
Antagonistic blending 364
Piepel direction 361
Scheffe polynomials 362
Simplex 360
Simplex centroid designs 365
Simplex lattice designs 363
Synergistic blending 364
Model diagnostics plots 337
Multi-linear constraint 357
Constrained designs 356
Constrained mixture spaces 371
Optimality 358
Optimisation designs 311, 316
Analysis 317
Box–Behnken 317, 318
Central composite design (CCD) 317, 349
Face centred design (FCD) 349
Which to choose 319
Resolution 307
Response optimisation
Graphical optimisation 353
Numerical optimisation 353
Response surfaces 342
Contour plots 342
Rotatable 318
Screening deisgns
Plackett–Burman designs 315
Screening designs 311, 313
Fractional factorial designs 307
Plackett–Burman designs 314
Special cubic 363
Star samples 317
Statistics
Adequate precision 328
Adjusted R-squared 328
Mean square 320
Mean squares of regression 325
Predicted R-squared 328
Pure error 334
Residual sum of squares 324
R-squared 328
Sum of squares regression 324
Total sum of squares 323
Steepest ascent 316
U-simplex 375
Design space 419, 429, 444
Desired state 419
Dimension reduction 79
Dimensions 166
Direct observations 2
Discrimination 9
Disjoint modelling 264
Dynamic time warping (DTW) 432
E
Early event detection (EED) 419
Edge of failure 429
Effective dimension 73
E-matrix 95
Empirical modelling 297
Engineering process control (EPC) 48
Estimate 161
Evolving factor analysis 403, 405
Evolving processes 432
Execution level 430
Expensive measurement 3
Experimental design. See Design of experiments (DOE)
Explained variance 95
Explained variance plot 144
Extrapolation 164
Extreme objects 151
F
Factor analysis 392
Factor matrix 392
Factor rotation
Equimax rotation 394
Parsimax rotation 394
Quartimax rotation 393
Varimax rotation 393
Orthogonal rotation 392
Factor influence study 316
Feature space 281
Fisher’s iris data 260
Fluid bed drying (FBD) 425
Focussed beam reflectance measurement (FBRM) 428
Formal statistical tests 30
F-residuals 150, 191
G
GAMP©5 431
Garbage in/Garbage out 138, 264
Gaussian 22, 51
General mixture models 362
Global estimation error (GEE) 58, 61, 223
Grab sample 56, 66
Grouping and segregation 247
Guided locally weighted regression 441
H
Half normal probability plot 322
Heterogeneity 15, 54, 223, 424
Heteroscedasticity 337
Hidden structures 6
Hierarchical cluster analysis (HCA). See Cluster analysis
Hierarchical modelling 255, 438
Classification–classification hierarchy 438
Classification–prediction hierarchy 438
Prediction–prediction hierarchy 438
Hierarchical replication 248
Histogram 15, 20
Homoscedasticity 337
Hotelling’s T<sup>2</sup> 149, 150, 151, 156, 191
I
Incorrect sampling errors (ISE) 221, 246
Increment 54, 66
Independent component analysis (ICA) 392
Oblique rotation 392
Independently and identically distributed (iid) 20
Indirect observations 2
Inexpensive measurement 3
Inferential statistics 14
Bonferroni limit 320
Confidence intervals 30
Empirical distribution function 31
Equivalence of means 38
False negative 27
False positive 27
F-test 35, 325
Hypothesis testing 26
<i>F</i>-test
F-ratio 320
Snedecor F-table 36
Kolmogorov–Smirnov 31, 51
Lilliefor’s correction 31
<i>t</i>-test
Paired <i>t</i>-test 196, 251, 453
<i>t</i>-tests 38, 51
Pooled standard deviation 41
Standard deviation 17, 18, 328
Variance 3, 17, 73, 75
Influence plot 150, 192, 205, 448
Inlier statistic 206, 453
In-line 422
Inner relationships 226
Interaction effects 302
International Conference on Harmonisation (ICH) 250, 419
Interquartile range 109
Iterative model development process 450
J
Joint confidence interval 50
K
Kernel functions 281
Kinetic modelling 403
Knowledge management system 431
Kurtosis 23
L
Lack of fit 93, 330, 334, 408
Lean manufacturing 431
Least squares fit 74
Leave-one-object-out. See Validation, Full cross-validation
Leverage 149, 150
Linear discriminant analysis (LDA) 279
Loading plots
Interpretation 83, 142
Loadings 77, 158, 179
Loading weights 179
Lot 65
Lot dimensionality transformation 60
M
Mahalanobis distance 150, 280
Manufacturing execution system (MES) 430
Manufacturing level 430
Martens’ uncertainty test. See Significance testing
Maturity index 433
Mean 14, 15, 328
Mean centre 77
Mean time before failure (MTBF) 429
Measurement error 35, 252
Measurement noise 113
Measurement uncertainty 53, 62, 66, 68, 245
Median 16
Mode 17
Model centre 77
Model diagnostics 190
Model lifecycle management 449
Modelling ability 164
Models 93
Interpretation 179
Model stability 210
Model updating 452
Model utilisation 438
Moving average 113
MS. See Mean square
Multiple linear regression (MLR) 167, 168, 169
Multiplicative 124, 129
Multiplicative effects 123
Multivariate calibration 161
Multivariate curve resolution (MCR) 398
Ambiguity
Intensity ambiguity 405
Rotational ambiguity 405
Constraints
Closure constraint 402
Estimated concentrations 408
Estimated spectra 408
Local rank constraints 402
Non-negativity constraint 401
Uni-modality constraint 401
Estimated concentrations 399
Estimated spectra 399
Multivariate curve resolution–alternating least squares (MCR-ALS)
405
Multivariate data analysis 1
Multivariate image analysis (MIA) 237
Multivariate statistical process control (MSPC) 50, 149, 150, 420, 429,
444
N
Negative challenge 286
Net analyte signal 187
NIPALS 94, 176
Noise 7, 122
Normal distribution 14, 20, 22, 322
Normal probability plot 24, 321
of residuals 337
Nugget effect 64
Null hypothesis 26, 268
O
Objectivity 257
Object residuals 78, 96
Object-to-model distance 268
Off-line 422
One-sample t-test. See Inferential statistics
One-sided 29, 36
One variable at a time (OVAT) 13, 299
On-line 422
Outliers 16, 139, 140, 141, 142, 146, 147, 148, 149, 150, 151, 153,
154, 156, 157, 158, 267
Example 154
Overall equipment effectiveness (OEE) 419
P
Paired t-test. See Inferential statistics
Parallel factor analysis (PARAFAC) 432
Pareto chart 320
Parsimonious model 209
Parsimony 258
Partial least squares regression 174
Partial least squares discriminant analysis (PLS-DA) 276
PLS1 175, 178
PLS2 175
Partial replication 248
Partition error 256
Pascal’s Triangle 306
PAT level 430
PCA. See Principal component analysis
PCA rotation. See Factor analysis
PC-models 93
PCR. See Principal component regression
PCs 73, 76, 97, 158
number of 76
Pharmaceutical quality management system (PQS) 418, 419
PLS. See Partial least squares regression
Polling rate 445
Population 13
Power 28, 42, 43
Precision 14, 35, 59
Intermediate precision 251
Repeatability 251
Reproducibility 251
Precision agriculture 417
Predicted residual sum of squares (PRESS) 196, 328
Predicted vs actual plot 338
Predicted vs reference 173
Prediction 10, 161, 162
Prediction ability 164
Prediction error 166
Prediction variance 166
Preprocessing 107, 130
Baseline correction 115
Correlation optimisation warping (COW) 127
Derivative
Second derivative 119
Derivatives 117
First derivative 117, 118, 119, 122
Quadratic baseline effects 127
Savitzky–Golay derivative 120
Second derivative 118, 119, 122, 128, 129
Segment-gap derivative 119
Detrending 126
Normalisation 114
Scaling
Logarithmic transformations 109
Mean centring 107
Median centring 109
Minimum centring 109
Spherising 109
Variance scaling 107, 108
Scatter correction
Common amplification model 125
Common offset 116
Common offset model 125
Extended multiplicative scatter correction (EMSC) 126
Multiplicative scatter correction 124, 144
Standard normal variate 123
Smoothing 109, 113, 129
Gaussian filter 113
Median filter 113
Moving block smoothing 113
Savitzky–Golay smoothing 114
Smoothing window 113
PRESS 196
Principal component analysis 69, 144
Algorithm 74
Summary 158
Principal component models 93
Principal component (PC) 73
number of 76, 94, 97, 166
Principal component regression 169
Principles of validation. See Validation
Probability density function 22
Problem-dependent 65
Problem reasons 218
Process analytical technology (PAT) 19, 418, 421
Process chronology 447
Process signature 435
Process validation guidance 420
Projection 151, 158, 447
Projection models 93
Projections 10
Proper process sampling 235
p-value 27, 28, 320, 325
Q
Q-residuals 191
Quality by design (QbD) 418
Quality risk management 419
Quality target product profile (QTPP) 419
Quantitative structure activity relationship (QSAR) 162
R
Random sampling 18
Range 109
Rank 70
Real time release 419, 442
Reference method 163
Regression 10, 161
Regression coefficients 113, 305
Relative coefficient of variation 61
Relative time modelling (RTM) 432
Remote sensing 423
Removing outliers 192
Replication 243, 245
Full replication 248
Replicate analysis 244
Replicate error 334
Replicate primary samples 246
Replicates 109
Replication experiment 61, 67, 246
Representative 163
Representative sampling 18, 66
Residual matrix 93
Residuals 74, 78, 95, 158, 408
Residuals vs predicted plot 337
Residuals vs run plot 338
Residual variance 95
Robust 16
Root mean
RMSECV 229
Root mean square errors 195
Root mean square error of cross validation (RMSECV) 195
Root mean square error of prediction (RMSEP) 166, 195, 232
Root mean square of calibration (RMSEC) 195
S
Sample 14, 65
Sampling bias 15
Sampling increment 56
Sampling unit operation 67
Sampling unit operations 54, 59
Savitzky–Golay 113
Scatter effects 122, 123
Scatter effects plot 130
Scatter plot 70
Scores 158
Projected scores 205
Score plots 80
Interpretation 80, 140, 142
Validation scores 152
Score trajectory 434
Score vectors 78
Self-modelling mixture analysis 399
Signal 7
Significance 26
Significance levels 268
Significance test 208
Significance testing
Martens’ uncertainty test 199, 208
Signs of problems 217
Simple structure 391
Singular value decomposition (SVD) 178
Singular values 405
Six Sigma 13, 431, 444
Skewness 17, 23
Soft Independent modelling of class analogy (SIMCA) 263
Coomans’ Plot 268
Discrimination power 265, 273
Membership plot 270
Model distance plot 272
Modelling power 276
Si/S0 vs Hi 272
Si vs Hi 270
Variable discrimination power plot 273
Span 164
Spatial lot coverage 58
Special cause 48
Specimen 65
Spectroscopy 87
Absorbance 111
Beer–Lambert’s law 111, 144
Diffuse reflectance 123
Full width at half maximum (FWHM) 121
Mid infrared (MIR) 111, 428
Molar absorptivity (ε) 111
Near infrared (NIR) 19, 110
Overlapping spectra 168
Raman spectroscopy 111, 426
Reflectance 111
Specular 123
Resolution 121
Spectroscopic data 110, 143
Transformations 111
Transmission 111
UV-Vis 111
Variables 107
Square error of prediction 195
SS 320
Stability plot 153
Stability test 208
Standard error 32
Standard deviation of differences (SDD) 43, 196, 251
Standard error of laboratory (SEL) 195, 251
Standard error of prediction (SEP) 197
Standard operating procedure (SOP) 138
Statistical methodologies 13
Statistical process control (SPC) 13, 47, 51, 444
Statistical sampling 222
Steady state processes 432
Studentisation 193, 337
Student’s t-test 149. See Inferential statistics
Subjectivity 257
Subject matter expertise (SME) 307
Sum of squares 18, 320
Supervisory control and data acquisition (SCADA) 430
Support vector machine classification 281
Support vectors 282
Suspicious data 157
Synergy 299
T
Theory of Sampling (TOS) 18, 53, 65, 221
Fundamental sampling error 247
Fundamental sampling principle 58, 67
Grouping and segregation error 59
Heterogeneity
Constitutional heterogeneity 54, 55
Distributional heterogeneity 54, 55
Spatial heterogeneity 245
Total analytical error (TAE) 58, 243, 449
Total analytical uncertainty 243
Total sampling error 58, 449
Unified sampling responsibility 68
Time series 45
Total process measurement system quality control 448
Total residual variance 96
Troodos 154
Tucker 3 models 432
Two-sample t-test. See Hypothesis testing
Two-sided 29
Type I error 27, 43, 268
Type II error 27, 43, 268
U
Uncertainty test 208
Univariate methods 1
Univariate regression 167
Univariate statistics 13
V
Validation 164
Cross-validation 165, 225
Categorical cross-validation 235
Full cross-validation 225
Leverage validation 165
Random cross-validation 234
Segmented cross-validation 225
Systematic cross-validation 235
Internal validation set 234
Principles of proper validation 221
Test set validation 164, 224
External validation set 234
Validation objectives 223
Variability 13, 14, 297
Variables
Category 310
Continuous 309
Dependent (y-variables) 161
Discrete variables 107, 310
Discrete numeric 309
Dummy 276
Independent (x-variables) 161
Indicator 310
Selective 6
Time series 107
Variable space 70
Variance
definition 3
Variographic analysis 63, 448
Nugget effect 67
Sill 67
Variography 67
Visualisation 255
V-model 453
X
X-residual 151, 190
x-variable. See Variables
X–Y relation outlier plots 197
Y
y-residual 193
y-variable. See Variables