Professional Documents
Culture Documents
Uncertainty Analysis of Measurement Results
Uncertainty Analysis of Measurement Results
M. Kostic
Department of Mechanical Engineering
At Ohio Aerospace Institute Lecture Hall Friday, August 1, 2003, 2:00 p.m.,
Refreshments Available at 1:30 p.m.
Presentation handouts: http://www.kostic.niu.edu/NASA-FFP03Kostic(D).pdf
Manuscript:
http://www.kostic.niu.edu/Uncertainty-Analysis-of-Measurement-Results.pdf References [ 1] Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results, NIST Technical Note 1297, 1994 Edition http://physics.nist.gov/Pubs/guidelines/TN1297/tn1297s.pdf , July 2003. [ 2] Galleries of Probability Distributions: NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/eda/section3/eda366.htm , July 2003. [ 3] R.H. Dieck, "Measurement Accuracy" a chapter in "The Measurement, Instrumentation and Sensors Handbook" (J.G. Webster, Editor-in-Chief), ISBN: 0-8493-8347-1, CRC Press, 1999. [ 4] NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/, July 2003. [ 5] Tables for Probability Distributions: NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/eda/section3/eda367.htm , July 2003. [ 6] R.S. Figliola, and D.E. Beasley, Theory and Design for Mechanical Measurements - 3rd Edition, Wiley, 2000. http://www.kostic.niu.edu/NASA/NASA-03Lecture.htm
categories or types [ 1]: The first type A uncertainties are due to random errors (also known as precision, or statistical uncertainties), which could be quantified using available statistical methods on a set of repeated measurements under apparently the same (or very similar) conditions. Additional uncertainties classification, type B uncertainties, are due to other than statistical errors associated to instrumentation and measurement procedure used (often unspecified, also known as biased or systematic errors). These, type B uncertainties, are evaluated using scientific judgment and previous related experience and calibration, including instrumentation specifications, other outside sources and references, and judgmental assumptions. Distinction between the two types of uncertainty (statistical type A and non-statistical type B) is not always clear, and meaning of many terms, like systematic error for example, may be misleading and should not be used exclusively, but only in relative terms. Systematic errors and specified biases if quantified (like after careful calibration) are no longer uncertainties, since they may be corrected, but only unspecified biases (due to unknown/random nature) are inherited uncertainties of instrumentation and similar, previous measurements. For example, a biased instrument uncertainty (when an instrument is used) is a type B uncertainty, based on previous calibration of that or similar instruments and usually obtained from outside sources; however, it was in part the type A statistical uncertainty during calibration and testing of that or similar instruments.
2. Evaluation/Quantification of Uncertainties
Even though uncertainty quantification is very judgmental and based on full physical understanding of all related issues and accumulated previous experience (it is more an art than a science), the uncertainty analysis procedure is rather simple, and consists of the following steps (see also an example below): (1) identification of all critical (i.e. the most contributing) uncertainty components (this is very important since the final result depends on what is accounted for); (2) quantification of all standard uncertainties, based on relevant statistics, like the Students t-distribution to account for small number of measurements (type A uncertainties), or obtained from other external sources (type B uncertainties); (3) use the law of propagation of uncertainties (sensitivity coefficients based on the firstorder Taylor series) when a partial or final measurement result is determined using an equation from other measured quantities; (4) reduce all uncertainty components to standard uncertainties or to uncertainties for the same probability confidence level (opposite from expended uncertainty, see below) and combine them using the Root-Sum-of-Squares (RSS) rule; (5) Expend the combined uncertainty to desired confidence level by multiplying it with a relevant coverage factor or dimensionless deviation t, based on the combined effective degree of freedom using the Student t-distribution. (6) Finally, report all relevant details of accounted uncertainty components along with expended resultant uncertainty, but also some indication of the values of uncertainty components not accounted for, should always be reported. The above steps will be detailed below. However, the final uncertainty results, regardless if the same analysis procedure is used on the same measured data, depends on the above step (1) and step (2) type B uncertainties, both being very judgmental and critical for the final results. Thus, the uncertainty of the result of a measurement is an estimate and can not have well-defined limits. Therefore, only a range or interval of possible errors may be quantified within some confidence level (probability), called measurement uncertainty, while absolute error (difference between measured value and true measurand) cannot be quantified and is a qualitative concept only.
However, the precision or random error (difference between a measured and their mean values) can be quantified. Many other terms, like precision, systematic error and bias, must be accurately defined and are often misleading. Fundamentally, statistics is based on large number of data, with an assumption that the uncertainty of the standard deviation is negligible. However, for a small number of measurements (often the case), not every estimated standard deviation is necessarily a standard uncertainty. In general, the measurement uncertainty depends not only on the repeatability (under the same measurement conditions) and reproducibility (under different conditions), but also on how well a measurement method has been chosen and implemented. As stated above, some indication of the values of uncertainty components not accounted for, should always be reported. Again, not only a measurement result, but also its uncertainty, are both uncertain within some confidence level.
xm =
1 n xi n i =1
n n xi xi i =1 i =1 = n( n 1)
n 2 2
(1)
Sx =
1 n xi xm ( 123 )2 = n 1 i =1
di
1 n 2 2 xi nxm n 1 i =1
(2)
Similarly, the standard deviation of data {xi, yi(xi)}, curve-fitted with a function ycf,i=ycf(xi), is calculated using the following correlation:
S xy =
cf
(y y
i =1 i
cf ,i
(3)
Note that Eq. (2) is a special case of Eq.(3) if yi =xi, ycf(xi)=xm and cf=n-1. The mean-value standard deviation, Sxm , is becoming smaller with increasing the number of measurements, n (thus n could be adjusted to achieve a desired mean-value uncertainty), and is expressed as:
S xm =
Sx = n
n 1 ( xi xm )2 n(n 1) i =1
( 4)
Assuming that a measurement uncertainty follows the normal Gaussian distribution (often very reasonable if measured data are bell-shaped and symmetric), it is expected that about 68.27% of all measurements (if the number is large enough) should be within Sx deviations from the mean value, while the remaining 31.73% are expected to fall outside the Sx deviations from the mean
value. The uncertainty probability (or confidence level) will increase to about 95.45% for 2Sx deviations from the mean value, and to 99.73% for 3Sx deviations from the mean value, according to the Gaussian distribution. The probability is uniquely related to the deviation range (i.e., uncertainty) for a given statistical distribution and vice versa, and may be determined from the corresponding tables or calculated using related functions in many standard application software, like Excel, MathCAD, MATLAB, etc., see also Table 1 and Refs. [ 1], [ 3], and [ 4]. If another set of measurements is repeated, with the same number of measurements, n, and under similar conditions as before, it is expected that the new mean value will be close to the previous mean value, actually that about 68.27% of all such mean values (if the number is large enough) will be within Sxm deviations from the mean of the mean values (overall mean), while the remaining 31.73% will fall outside the Sxm deviations from the overall mean value (remember that S xm = S x / n < S x ). The uncertainty probability will increase to about 95.45% and 99.73% for 2Sxm and 3Sxm deviations from the overall mean value, respectively. If number of measurements is not large enough (say n<20 or so, which is often the case), it is more accurate to use the Students t-distribution instead of the normal Gaussian distribution. The concept and procedure is the same except that values of probability are dependent, in addition to the deviation range, also on degree of freedom (), which in turn is dependent on number of measurements (for a measured constant value the =n-1). For the same deviation, for example, for Sx deviations from the mean value, the probability of the Students t-distribution will increase with number of measurements from 50% for n=2 (i.e. =1) to 65.66% for n=10, and will approach the Gaussians 68.27% probability for a large number of measurements. The Students t-distribution tables and functions are available along with normal Gaussian and other statistical functions in many standard references [ 5] and software. However, the relevant data are usually presented in dimensionless forms (to avoid any issues related with units), and dimensionless deviations or intervals (sometimes referred to as coverage factors) are defined as:
tk = zk =
d k xk xkm = Sk Sk
(5)
The statistical parameters are mutually and uniquely correlated for different statistical distributions, for example:
(6) (7)
where t (or z, see Eq. 5) is the Student t-distribution interval, a statistical function of degree of freedom , and confidence level P%, which uniquely correlates the three quantities (Eq. 6), and it reduces to the normal Gaussian correlation for infinite (or large enough) degree of freedom (Eq. 7), see also Figure 1. From implicit correlations, Eqs. (6 or 7), any parameter may be explicitly expressed as a corresponding statistical function of the remaining parameters, available in standard software or tabulated in references. Different symbols, zk and tk, for otherwise same quantity (and concept), namely dimensionless deviation, also called the coverage factor, are used to indicate if the Gaussian or Students t-distributions are used, respectively. As stated before, the probability for a given statistical distribution and number of measurements (i.e. degree of freedom) is dependent on the deviation magnitude only and vice versa. Both the Gaussian and Students t-distributions are symmetrical around the mean value (or zero dimensionless deviation) and integral over each side is 50%, to make up the total probability of 100% for all deviations possible (when the dimensionless deviation ranges from negative to positive infinity). In different references (and software) different portions (fractions) of probabilities are given for the same
dimensionless deviation which may introduce unnecessary confusion. However, those different portions of probabilities are simply correlated and thus conversion is straightforward when required, see Figure 1 and 0.4 Table 1. Either an inside Gaussian or dP/dt probability, Pin, is given for Pin+ Pout=1=100% Students positive deviation range, or t-distributions 0.3 double inside, symmetrical probability, Pin, is given. Also Pin Pin corresponding outside, singletailed probability, Pout=50%0.2 Pin, or double-tailed outside Pt,z=0.5+Pin P(t)= Pin probability, Pout=100% -Pin, may be given. Furthermore, so 0.1 called quantile or cumulative Pout Pout probability for a deviation range from negative infinity to an arbitrary value is given, for 0 example in MathCAD, and it is -4 -3 -2 -1 1 2 3 4 0 t or z t=z related to one-sided inside -t=-z probability as: Figure 1: Relation between uncertainty (t or z) and corresponding probability (P) Pz=Pupto_z=0.5+sign(z) Pin|z|. Since Students t-distribution accounts for (small) number of measurements and reduces to normal Gaussian distribution for large number of measurements, it is recommended to be used always, because it accounts for the uncertainty of the standard deviation itself, the latter assumed to be negligible for large number of measurements when the Gaussian distribution may be used. Given, in Table 1 below, are the corresponding Excel functions and formulas along with nomenclature introduced here, to calculate important statistical parameters, some of them available in Excels Descriptive Statistics in Data Analysis submenu within the Tool menu (if add-on Analysis ToolPak is installed, see the last column in Table 1). Therefore, for a desired probability, P%, the dimensionless deviation, t, is evaluated first, using relevant tables or software (like in row 16 in Table 1 below), and then the dimensional deviation or uncertainty is calculated using the corresponding sample- (or mean-, if appropriate) standard deviation, i.e.:
d ,P % = u ,P % = t ,P % S x ( m )
(8)
Conversely, for a given or desired dimensional deviation d (i.e. uncertainty u), and degree of freedom, =n-1, the dimensionless deviation, t, is evaluated first, using the corresponding sample (or mean-value standard deviation if appropriate), i.e.:
t ,P % =
d ,P % u ,P % = S x(m) S x(m)
(9)
and then the corresponding (double-sided-inside) probability, P,t, is evaluated using relevant tables or software (like in row 17 in Table 1 below).
It is straightforward to recalculate a known (old) deviation or uncertainty at one (old) given probability to a new deviation or uncertainty for another (new) desired probability or vice versa, as described in Section 2.5. Table 1: Statistical parameters and related Excel functions
Row # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 u,P%=t,P%Sxm Pt, n Sx Sx
2
Excel functions and formulas AVERAGE(Data) STDEV(Data)/SQRT(COUNT(Data)) MEDIAN(Data) MODE(Data) STDEV(Data) VAR(Data) KURT(Data) SKEW(Data) MAX(Data)-MIN(Data) MIN(Data) MAX(Data) SUM(Data) COUNT(Data) LARGE(Data, i_th) SMALL(Data, i_th) TINV(1-Probability, COUNT-1)*Standard_Error 1-TDIST(t, COUNT-1,2)
Excel Descriptive Statistics items Mean (average) Standard Error (of mean) Median (mid-way of data) Mode (most frequent data) Standard Deviation (of data sample) Sample Variance Kurtosis (peakedness) Skewness (asymmetry) Range Minimum Maximum Sum Count (number of data) Largest(i_th=1) Smallest(i_th=1) Uncertainty at Confidence Level (for 95.0%=Probability) Probability (Double-Inside) for given deviation t
old bias or systematic uncertainties. Intrinsically, the errors and uncertainties are random and statistical, and only distinction is whether the random scatter is observed and processed by an experimenter at present time (type A) or inherited from others long time experiences (type B uncertainty, passed to us from previously experienced and accumulated type A uncertainties by others). The instruments type B uncertainties are usually given by manufacturers or estimated and given in references for a well-known class of instruments manufactured within standard specifications. The most critical is estimation of type B measurement methods uncertainties, since they depend on the employed methods of measurements and care of relevant details. For example, if we are manually timing a process with a stop-watch of 1 millisecond of instrument uncertainty, but starting and stopping synchronization process is uncertain within several tenths of a second, then the latter uncertainty is more contributing to the result than the former one. That why the knowledge and experience is more critical for proper quantification of the type B than type A uncertainties, since the former ones are less obvious. Generally speaking, the type B uncertainty quantification depends on the available external information and the experience and assumption of an experimenter.
2.3. Sensitivity or Dependence of Result on Other Quantities: The Law of Propagation of Uncertainties
If a measured result, xR, is a function of (i.e. depends on) other measured quantities, xi, with respective uncertainties ui,
xR = f ( x1 , x2 ...xi ..., xN )
(10)
then, there will be an uncertainty in result due to each of uncertainties of measured quantities xi, i.e.:
u R ,i =
f ui = ci ui ; (1 < i < n) xi
(11)
The above is a partial uncertainty in result uR,i, due to dependence on a measured quantity xi, and its uncertainty ui. The dependence or sensitivity of the measured result on the measured quantity xi, is expressed by magnitude of the corresponding partial derivative (based on the first-order Taylor series coefficients) and is also known as sensitivity or dependency coefficient, ci. Regardless of the different dimensions of the measured quantities xi, and their uncertainties ui, after multiplication with the corresponding partial derivatives (sensitivity coefficients) they will all result in the same dimension of the resultant quantity and could be combined together (see Section below). If a particular partial derivative has a zero value, it means that the result is not dependent on that particular measured quantity, the latter uncertainty will not propagate and affect the resultant uncertainty. Conversely, if the magnitude of the partial derivative (thus dependence or sensitivity) is very large, then that component uncertainty will propagate very strongly (will amplify) and may adversely influence the resultant uncertainty.
uc = u R ,c = u R ,1 + u R , 2 + ... + u R , N =
uR,i
i =1
f ui = = i =1 xi
N
(c u )
i =1 i i
(12)
c = R ,c
i =1
Note that i (or c) for type B uncertainty is usually very large (all time history) and may be taken as infinity if not specified otherwise, i.e. the Gaussian distribution may be used. It is important that every component of uncertainty, uR,i, due to different source of uncertainty be mutually uncorrelated (zero covariance), be at the same confidence level (probability), and of course they all should have the same dimension expressed in the same units of measurement. Every uncertainty component may be estimated independently (thus, ci=1), or determined using the law of propagation of uncertainties, Eq. (11), if the source of uncertainty is due to sensitivity (dependence) on measured component quantity (ci=|f/xi|).
unewP % =
(13)
where t,newP% is the Student t-distribution interval (also called the coverage factor), a statistical function of degree of freedom (or c if appropriate), and confidence level P%, which uniquely correlates the three quantities and reduces to the normal Gaussian correlation for infinite (or large enough) degree of freedom, see Eqs. (6 & 7). This conversion will be needed for all types of uncertainties (if available at different probability confidence level) to bring them to the same probability level, before combining them using the root-sum-of-squares (RSS) procedure, Eq. (12), described above. If doldP% (or uoldP%) = Sx(m) then t,oldP%=zoldP%=1, i.e.:
(14)
Also, for any desired deviation interval, dnewP% (or unewP%), the corresponding confidence level (i.e. probability), P%, may be determined using relevant statistical tables or formulas. It will be necessary to first determine the corresponding dimensionless deviation, t or z, based on given standard deviation, Sx(m), or on a known uncertainty interval at a known confidence level, uoldP%, i.e.:
tnewP % =
(15)
t ,newP % =
(16)
Equation (15) is a special case of Eq. (16) when uoldP%=Sx(m) since then t,oldP%=zoldP%=1. Conversely, for a desired confidence level P%, the corresponding dimensionless deviation or uncertainty may be determined using relevant statistical tables or formulas, and then dimensional deviation or uncertainty determined from Eqs. (13 & 14).
uany , r =
uany xany
uany xany
100%
(17)
It is recommended and often advantageous to evaluate the corresponding relative uncertainty, and particularly the extended (resultant) relative uncertainty, in order to evaluate its relative fraction or percentage of the measured quantity. The uncertainty of a well-conducted measurement, by its very nature, is supposed to be small, a couple or several percent. Negligibly small percentage may indicate that not all critical sources of errors were accounted for, and too large percentage may indicate that an inaccurate instrument or inappropriate measurement methods were employed. Any extreme results should be reviewed and appropriate actions taken.
P = P(V , R) = V 2 / R
( 18)
uP =
(u ) + (u )
2 P ,V P, R
2 P P 2V V = uV + u R = uV + 2 u R R V R R
( 19)
However, if resistance is a known function of temperature T, say RT=R0[1+b(T-T0)], and the temperature T and voltage V are measured by an experimenter, as well as an additional measurement bias, B, is empirically estimated, then the power measurement equation (which accounts for all major contributing components) becomes:
P = P(V , R0 , b, T , T0 , B) =
V2 +B R0 [1 + b(T T0 )]
( 20)
The uncertainty of the power is then determined using sensitivity and RSS Eqs. (11 & 12), as well as known or estimated uncertainties of all components (uV, uRo, ub, uT, uTo, uB), i.e.:
uP =
(u ) + (u ) + (u ) + (u ) + (u ) + (u )
2 2 2 2 2 P ,V P , Ro P ,b P ,T P ,To P, B
2 2 2 2 P 2 P P P P P uV + u R + ub + uT + uT0 + u B = V R0 0 b T T0 { B 0
( 21)
Note that B may have any value, including zero, still its uncertainty may exist, and the corresponding sensitivity P/B=1. Uncertainties much smaller that the others may be neglected (like uTo0). Details and evaluation of uncertainty components of a resultant uncertainties may differ, but if all critical sources of errors are properly accounted for, then the final results should be close to each other when expended to the same confidence level.
4. Conclusion
As already stated, the uncertainty quantification is very judgmental and based on full physical understanding of all related issues and accumulated previous experience (it is more an art than a science), however, the uncertainty analysis procedure is rather simple, and consists of the following steps: (1) identification of all critical (i.e. the most contributing) uncertainty components (this is very important since the final result depends on what is accounted for); (2) quantification of all standard uncertainties, based on relevant statistics, like the Students t-distribution to account for small number of measurements (type A uncertainties), or obtained from other external sources (type B uncertainties); (3) use the law of propagation of uncertainties (sensitivity coefficients based on the firstorder Taylor series) when a partial or final measurement result is determined using an equation from other measured quantities; (4) reduce all uncertainty components to standard uncertainties or to uncertainties for the same probability confidence level and combine them using the Root-Sum-of-Squares (RSS) rule; (5) Expend the combined uncertainty to desired confidence level by multiplying it with a relevant coverage factor or dimensionless deviation t, based on the combined effective degree of freedom using the Student t-distribution. (6) Finally, report all relevant details of accounted uncertainty components along with expended resultant uncertainty, but also some indication of the values of uncertainty components not accounted for, should always be reported.
10
It is important to clarify again, that only the uncertainty, i.e. a deviation range (or interval) from a reported measurement result, with corresponding probability, may be evaluated, but it is not possible to obtain a perfect (error-free) measurement, nor it is possible to estimate an uncertainty with 100% probability (absolute certainty is impossible too). However, under well-controlled conditions and well-understood measurement process and procedure, it is possible to minimize and reasonably well (at high probability) estimate the uncertainties of measured quantities and the final measurement result.
References
[ 1] Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results, NIST Technical Note 1297, 1994 Edition http://physics.nist.gov/Pubs/guidelines/TN1297/tn1297s.pdf , July 2003. [ 2] Galleries of Probability Distributions: NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/eda/section3/eda366.htm , July 2003. [ 3] R.H. Dieck, "Measurement Accuracy" a chapter in "The Measurement, Instrumentation and Sensors Handbook" (J.G. Webster, Editor-in-Chief), ISBN: 0-8493-8347-1, CRC Press, 1999. [ 4] NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/, July 2003. [ 5] Tables for Probability Distributions: NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/eda/section3/eda367.htm , July 2003. [ 6] R.S. Figliola, and D.E. Beasley, Theory and Design for Mechanical Measurements - 3rd Edition, Wiley, 2000.
11