Professional Documents
Culture Documents
Six Sigma Metric Analysis For Analytical Process
Six Sigma Metric Analysis For Analytical Process
Introduction
Laboratories seek objective assessment and comparison size of the quality requirement. Where the arrow hits that
of analytical methods and instrumentation performance. target is determined by the method performance data.
Unfortunately, there are few ways to compare systems on a
level playing field to make an “apples to apples” comparison.
Current methods of assessment can be arbitrary, relying on
unclear “state of the art” assessments, or focusing more on
easily tangible efficiency metrics, such as speed, cost, or
ease of use. Analytical goals and requirements for the
quality delivered by a test are often overlooked during
the decision-making process leading to the purchase of
instrumentation. Rapidly changing regulatory schemes
increase the confusion over acceptable standards for
instrument and method quality.
MS-09-7907 v4.0
Adopting Six Sigma as the Goal for Laboratory Testing
Six Sigma is a widely-accepted quality management system, – many laboratories use them for all testing processes.
perhaps best known outside of healthcare as the product The misuse of 2s limits in laboratory testing frequently results
of innovation at General Electric and Motorola.1 Six Sigma in erroneously-repeated controls, excessive trouble-shooting,
is also well known for the colorful titles of its practitioners – or worse still, workarounds that artificially widen control limits
green belt (part-time Six Sigma worker), black belt (full-time to the point that laboratories can no longer detect critical
Six Sigma worker), master black belt (consultant to black analytical errors.
belts), and champion (executive proponent of Six Sigma
efforts). Six Sigma has been adopted by both manufacturing Part of the power of the Six Sigma scale is its ability to
and service industries, as well as healthcare institutions from provide a universal benchmark. Sigma metrics allow
hospitals to reference laboratories. comparison of different processes with each other, even
comparing processes across different institutions and
Six Sigma is a metric that quantifies the performance of different industries. For example, airline safety is known to
processes as a rate of Defects-Per-Million Opportunities, be better than Six Sigma with a rate of only 1.5 crashes per
(DPM, or DPMO). Six Sigma programs also encompass million departures, while airline baggage handling in the
robust techniques such as Define-Measure-Analyze- U.S. is only 4.1 Sigma since approximately 1% of luggage
Improve-Control (DMAIC) and Root Cause analysis to find is misplaced or lost, and U.S. airline departures perform
and eliminate defects and variation within a process. at only 2.3 Sigma since nearly 30% of flights are delayed,
which helps to explain chronic customer complaints.
The goal of Six Sigma, in its simplest distillation, is to
eliminate or reduce all variation in a process. Variation In healthcare, the Sigma performance of common
in a process leads to wasted effort and resources on processes are less well known. When the Institute of
retesting and workarounds for example. Reducing Medicine issued its landmark report, To Err is Human,2
defects reduces costs, and improves performance it famously revealed that between 48,000 and 90,000
and profitability. A process that achieves the goal of unnecessary deaths occurred in U.S. hospitals every year.
Six Sigma delivers both quality and efficiency. Examining the death rates at the hospitals that formed the
basis of the study reveals that healthcare is performing at
The quantitative goal of Six Sigma is to create a process that only 3.8 Sigma. If healthcare were achieving Six Sigma,
minimizes variation until six standard deviations can fit within the death rate would be only 16 to 34 deaths per year.
the tolerance limit (Figure 2). At the level of Six Sigma
performance (world class quality performance), approximately Nevalainen’s3 groundbreaking work in Sigma assessment
three defects will occur per million opportunities. in the clinical lab analyzed the performance of common
laboratory processes and found that many were woefully
- TEa + TEa inadequate:
Bias
True Value
Figure 2: Relationship of imprecision (CV), inaccuracy (Bias) Chemistry specimen acceptability 0.30 3,000 4.25
and allowable total error (TEa) in predicting defects Surgical pathology specimen accessioning 3.4 34,000 3.30
The Six Sigma scale typically runs from zero to six, but Cytology specimen adequacy 7.32 73,700 2.95
a process can actually exceed Six Sigma, if variability is Laboratory proficiency testing 0.9 9,000 3.85
sufficiently low as to decrease the defect rate. In industries Surgical pathology frozen section
1.7 17,000 3.60
outside of healthcare, 3 Sigma is considered the minimal diagnostic discordance
acceptable performance for a process. When performance PAP smear rescreening false negatives 2.4 24,000 3.45
falls below 3 Sigma, the process is considered to be
Reporting errors 0.0477 477 4.80
essentially unstable and unacceptable.
Figure 3: Sigma metrics of common laboratory processes, Nevalainen data.
In contrast to other industries, healthcare and clinical
laboratories appear to be operating in a 2 to 3 Sigma Revisiting the arrow and target model, Six Sigma provides the
environment. The routine use of “2s” (i.e., 2 standard shape of the target. The shape defines the goal of Six Sigma
deviations or 2 SD) control limits is indicative of a complacent performance as the bull’s eye, as well as the inner rings of
tradition in quality control practices. Despite the well-known 4 and 5 Sigma. Outside the 3 Sigma ring, performance is
problems of 2s limits – they can generate false rejection rates considered to have missed the target and the process is not
of up to 10 to 20%, depending on the number of controls run considered to be “fit for purpose.”
MS-09-7907 v4.0
Defining Quality Requirements
Knowing the shape of the target is not enough. The size Clinical benchmarks can also be used to generate quality
must also be described. In Six Sigma terminology, the requirements. Dr. Carmen Ricos and her colleagues have
tolerance limits must be defined. In the clinical laboratory, provided a continuously updated database of biologic
the quality required by an analytical testing process must variation since 2000. For over 300 different analytes, they
be defined. Tolerance limits, in the laboratory, are best have tabulated desirable specifications for imprecision,
expressed as a total allowable error (TEa) specification. inaccuracy, and total allowable error.5 ISO 15189, the
new international lab accreditation standard for quality in
TEa is a well-accepted concept in healthcare laboratories, laboratories, also provides guidance on analytical testing.
as a model that combines both the imprecision and the Finally, the growing body of research on Evidence-Based
inaccuracy (bias) of a method to calculate the total impact Laboratory Medicine (EBLM) Guidelines can be used to
on a test result.4 An allowable total error is the expression develop Clinical Decision Intervals. These intervals can,
of how much combined imprecision and inaccuracy can in turn, be used to determine quality requirements for
be tolerated in the test result without negatively impacting individual tests. At the very least, a laboratory can
patient care based on interpretation of that result. consult the clinicians who use its test results and, by
documenting how test results are interpreted, determine
Determining the quality required by a laboratory test is not
the quality required by their testing processes.
as simple as it sounds. Most laboratories do not know the
analytical quality required by their tests. Indeed, many Establishing quality requirements, returning to the arrow
laboratories assume that it is not even necessary to know. and target model once more, determines the size of the
As long as there are no direct complaints about testing target. Since the use and performance of different tests
quality, many laboratories assume that the analytical varies, so too does the size of the target that the
quality they are providing is adequate. This is not the only arrow / process must hit. Together, Six Sigma and Quality
crippling assumption that laboratories make. Sometimes, Requirements provide the shape and size of the target.
laboratories assume that the quality of any test is sufficient Now all that remains is determining where (and if) the
simply because a manufacturer built the instrument and arrow hits the target. For that, we need data on the
made the reagents. While it’s common to assume that no actual performance of the process.
manufacturer would produce instruments and reagents that
perform poorly, it is not good laboratory practice. Finally,
laboratories frequently assume that simply following the
manufacturer’s directions is enough to assure the quality
of the tests they provide. Again, the fact that a manufacturer
provides directions does not guarantee that the directions
are adequate. Professional standards as well as regulatory
requirements place the burden of selecting appropriate
quality control procedures on the laboratory, specifically
the laboratory director.
MS-09-7907 v4.0
Measuring Six Sigma Performance in the Laboratory
Usually, Sigma performance is assessed by counting Ideally, the data on imprecision and inaccuracy is collected
defects, then by converting that count into a Defects Per during the same time frame and at the same critical level
Million Opportunities (DPM, or DPMO) rate. Once the DPM (medical decision level) of test interpretation. In other words,
is known, a Six Sigma table, available in standard text- the data on performance should be an accurate snapshot
books, can be consulted to obtain the Six Sigma metric. of method performance at a specific point in time and at a
specific concentration of analyte. Thus the resulting Sigma
Counting defects relies on two capabilities. First, it must be metric best reflects actual test performance. For example, if
possible to define what a process defect means. Second, the critical level is in the lower end of the dynamic range, a
it must be possible to detect a process defect when it bias estimate should also be obtained from the same
occurs. For most processes, these are simple tasks. Most concentration range, or the regression equation from a
processes analyzed in Six Sigma projects use the counting comparison of methods study can be used to estimate
defect approach. bias at the critical level. For tests with multiple critical
levels, it may be desirable to make Sigma metric estimates
In the laboratory, counting defects is also the usual Six
at each level.
Sigma metric technique. For example, turn-around time
(TAT) is very easy to define for laboratories. A laboratory The relationship of imprecision and inaccuracy to Sigma
might set a target of returning test results within 60 minutes metrics can be graphically depicted (Figure 2). Given a
of specimen receipt. Thus, when a test result is returned normal distribution of test results, as well as a known
after 61 minutes, it’s simple to detect the defect (i.e., TAT standard deviation (the imprecision) and a known bias, the
is > 60 minutes). Counting the number of defective test acceptable performance range can easily be calculated
results (> 60 minutes) over a period of time is an easy and, conversely, the concentration ranges in which results
way to determine the Sigma performance of the are unacceptable can also be defined (i.e., the concentration
laboratory’s TAT. ranges above and below the tolerance limits that define TEa).
For laboratory test results, however, determining and The relationship between imprecision and inaccuracy to
detecting defects is more difficult. When a single test result Sigma metrics can be summarized mathematically by the
is generated, it’s not possible to know what the true value following equation:
of that test result should be, even if the sample is tested
multiple times. For example, if a cholesterol test result is Sigma metric = (TEa – bias observed) / CVobserved
212 mg/dL, the “true value” of that test is not known,
unless the specimen was also analyzed by an accepted A simple example with the Sigma-metric equation reveals
reference method. Thus, it’s unknown if the result falls that the “state of the art” in healthcare is not Six Sigma. For
within the tolerance limits or quality requirements. If the cholesterol, CLIA defines an allowable total error of 10%.
true value is 190 mg/dL, the observed test result is That is, a cholesterol test result must be within 10% of its
probably a defect. If the true value is 205 mg/dL, the true value. The National Cholesterol Education Program
observed test result is probably acceptable. But without (NCEP) established separate goals for imprecision and
knowing the true value, there is no way of counting how inaccuracy of 3% each. A method that performs with 3%
many defects are being generated by a testing process. CV and 3% bias is considered acceptable by the NCEP.
The Sigma metric calculations tell another story:
Fortunately, there is another method of determining the
Sigma metric of a process: by measuring variation. (10 – 3) / 3 = 2.33 Sigma
Conveniently for laboratories, measuring variation through
the use of controls is part of the daily routine. Controls are a This is stark proof that laboratories currently operate in an
known value, so variation of an observed test result can be environment in which world class performance is not the goal.
measured. With multiple control results, information on the
standard deviation of testing processes can be collected Returning once again to the arrow and target model, the
and the imprecision (coefficient of variation, % CV) can be Sigma approach gives us a target, the quality requirement
calculated. Information about the inaccuracy (bias) of an gives us the size of that target, and the performance data
analytical testing process can readily be calculated by of the method give us the arrow, which should land as
comparing results between the testing method and a close to the bull’s eye (Six Sigma) as possible.
reference method, or by analyzing the results of the testing
method in proficiency testing, peer group, or some other
form of external quality assurance program.
MS-09-7907 v4.0
Measuring Six Sigma Performance in the Laboratory (continued)
Even when data on method bias is missing, a modified Sigma Normalized Method Decision Chart
metric can be calculated. The resulting metric documents the 100
Ex
20
cel
Ma
World Class
Go
len
rg
Sigma metric capability = (TEa) / CVobserved 10
Po
1
od
in
o
t
r
6 5 4 3 2
al
0
0 10 20 30 40 50
The performance of methods can be graphically illustrated
Observed Imprecision, % CV
using a Method Evaluation Decision chart 6 (MEDx, Figure
4) with Six Sigma metric lines imposed upon them. The Figure 5: Normalized Method Decision chart
Method Decision chart displays inaccuracy on the y-axis,
imprecision on the x-axis. Typically the chart is drawn for Normalized Method Decision Chart
each specific quality requirement (i.e., a 10% quality 100
30
For example, if a test had a quality requirement of 10% and a
CV of 1% and a bias of 2%, the coordinates on a Normalized 20 Ex
cel
Ma
World Class
Method Decision chart would be (10, 20) (Figure 6). Go
len
rg
10
Po
1
od
o
t
na
r
6 5 4 3 2
l
0
Method Decision Chart TEa=10% 0 10 20 30 40 50
10.0 Observed Imprecision, % CV
TEa
9.0
Figure 6: Normalized Method Decision chart with sample data point
8.0
Un
Observed Inaccuracy, % Bias
7.0
ac
ce
While Normalized Method Decision charts with Six Sigma
pt limits incorporate many complex features and calculations
ab
6.0 le
into a single display, the result of the chart still fits within
5.0 the arrow and target model. The chart can be visualized
as the upper right quadrant of the target. The area around
Po
4.0
or
Ma
the origin (0,0) of the chart (and below all of the lines) is the
Go ellent
Ex
rg
3.0
c
od
in
bull’s eye. The Sigma lines drawn on the chart are similar to
al
2.0
World Class the rings of the target, with 3 Sigma representing the edge
1.0 TEa/6 TEa/5 TEa/4 TEa/3 TEa/2 of the target (anything below 3 Sigma is considered off the
0 target, i.e., unacceptable). The x- and y- coordinates of a
0.0 1.0 2.0 3.0 4.0 5.0
plotted test represent the performance of the test and the
Observed Imprecision, % CV
spot where the arrow “landed.”
Figure 4: Method Decision chart for an allowable total error of 10%
MS-09-7907 v4.0
Benefits of Sigma Assessment
Given the simple parameters of the Sigma-metric equation, provided by the manufacturer upon request. With that
laboratories can easily determine the current performance data, laboratories can calculate Sigma metrics, compare
of all their current methods. The data acquired during the them to the Sigma metrics of competing instruments, and
standard method validation protocols of a new instrument use this tool as part of their decision-making process. This
can also be used to determine performance metrics. In application, to objectively assess and compare instrument
addition, formal method validation studies are typically performance before purchasing a new instrument, is of
conducted on all new methods and are often presented considerable value. It gives the laboratory the power to
as posters at scientific conferences, or as more scholarly predict which methods will perform to their clinical needs
reports published in scientific journals, or simply as data and which will not.
One Step Further: From Sigma Metric Assessment to Quality Control Design
OPSpecs Chart TEa=10% with 90% Error Detection
Pfr N R
10.0
13s/2of32s/R4s/31s/6x
9.0
0.07 6 1
13s/22s/R4s/41s/8x
8.0
0.03 4 2
Allowable Inaccuracy
7.0 13s/22s/R4s/41s
0.03 4 1
6.0 12.5s
3.
00
4.0 0.03 2 1
pe
13s/22s/R4s
rf o
3.0
rm
0.01 2 1
an
13s
ce
2.0
0.00 2 1
1.0 13.5s
0.00 2 1
0.0
0.0 1.0 2.0 3.0 4.0 5.0
Allowable Imprecision (s, %)
Sigma metric analysis is not confined to the role of highest error detection (thus, there is more “room” beneath
assessment and method validation. Sigma metrics can also that line for the method to hit). Other details about the QC
be used to refine and streamline the operational routines of procedure are noted in the key on the right side of the
a method. Combining Sigma metrics with QC Design tools, chart, such as the false rejection (Pfr) and number of control
such as the Operating Specifications chart (OPSpecs), measurements (N) and the number of runs (R). The
allows the laboratory to customize and optimize the QC imprecision and inaccuracy of a method are used as the
procedures conducted by the laboratory. A rational QC x-coordinate and y-coordinate, respectively. If this
Design can eliminate much if not all of the wasteful 2s QC “operating point” lies below one of the lines of the OPSpecs
practices, replacing them instead with appropriate control chart, that indicates that the QC procedure represented by
limits and numbers of control measurements. that line will provide the appropriate performance (Figure 7).
An OPSpecs chart provides a graphic description of the Reprising the arrow and target model one final time, the
imprecision and inaccuracy that are allowable and the OPSpecs chart can be viewed in the same way as the
control rules and number of control measurements that are Sigma metric analysis and the Method Decision chart.
necessary for a QC procedure to achieve an appropriate The OPSpecs chart is like the upper right quadrant of the
level of analytical quality assurance for a defined quality target, with the origin as the bull’s eye. Method perfor-
requirement.7 The diagonal lines in this chart represent mance (the arrow) should be as close as possible to the
the error detection performance of actual QC procedures bull’s eye. This time, however, the different rings on the
(control rules and numbers of measurements). These lines target represent different QC procedures for use in the
are arranged from top to bottom according to their error laboratory. The closer to the bull’s eye, the more QC
detection capability; the highest line provides the procedures available for quality management.
MS-09-7907 v4.0
One Step Further: From Sigma Metric Assessment to Quality Control Design (continued)
OPSpecs charts, as with Method Decision charts, are typically generated for specific quality requirements. But OPSpecs
charts can also be normalized so that multiple tests with different quality requirements can be displayed on the same
chart (Figure 8).
80.0 P N R
%)
fr
meas
70.0 1
2.5s
3.
00
0.03 2 1
Allowable Inaccuracy (bias
60.0
sig
1 /2 /R
3s 2s 4s
m
a
0.01 2 1
pe
50.0
rf o
1
3s
rm
an
40.0 0.00 2 1
ce
1
3.5s
30.0
0.00 2 1
1
20.0 3s
0.00 1 1
10.0 1
4s
0.00 2 1
0.0
0.0 10.0 20.0 30.0 40.0 50.0
Allowable Imprecision (s %)
meas
Figure 8: Sample Normalized OPSpecs chart
Conclusion
Sigma metric analysis, Method Decision charts, and OPSpecs charts provide easy tools for laboratories to determine
the performance of their current methods and QC design, and to compare competing instruments on the markets. Both
quantitative calculations and visual assessment can be made with this approach. These techniques give the laboratory
a practical way to select the right method and then select the right QC for that method. The result is an optimized testing
process that fulfills the quality required for appropriate test interpretation.
MS-09-7907 v4.0
1 Westgard JO, Six Sigma QC Design and Control, 2nd Edition, Madison WI: Westgard QC, 2006.
2 Institute of Medicine. To Err is Human: Building a Safer Health System. Washington DC: National Academy Press, 2001.
3 Nevalainen D, Berte L, Kraft C, Leigh E, Morgan T. Evaluating laboratory performance on quality indicators with the six sigma scale. Arch Pathol Lab Med 2000;124:516-519.
4 Westgard JO, Basic Planning for Quality, Madison WI: Westgard QC, 2000
5 Ricos C, Alvarez V, Cava F, Garcia-Lario JV, Hernandez A, Jimenez CV, Minchinela J, Perich C, Simon M. Desirable specifications for Total Error, Imprecision and Bias derived from Biologic
Variation, http://www.westgard.com/biodatabase1.htm Accessed December 12, 2008.
6 Westgard JO. A Method Evaluation Decision Chart (MEDX chart) for judging method performance, Clinical Laboratory Science, 1995, vol. 8, no. 5, pp. 277-283
7 Westgard JO, Charts of operational process specifications (“OPSpecs charts”) for assessing the precision, accuracy, and quality control needed to satisfy proficiency testing performance
criteria Clinical Chemistry 38: 1226-1233, 1992
MS-09-7907 v4.0