14.3: Validating the Method as a Standard Method

14.3.1 Two-Sample Collaborative Testing

The design of a collaborative test must provide the additional information we need to separate random errors from the systematic errors introduced by the analysts. One simple approach—accepted by the Association of Official Analytical Chemists—is to have each analyst analyze two samples that are similar in both their matrix and in their concentration of analyte. To analyze their results we represent each analyst as a single point on a two-sample chart, using the result for one sample as the x-coordinate and the result for the other sample as the y-coordinate.⁸

As shown in Figure 14.19, a two-sample chart divides the results into four quadrants, which we identify as (+, +), (–, +), (–, –) and (+, –), where a plus sign indicates that the analyst’s result exceeds the mean for all analysts and a minus sign indicates that the analyst’s result is smaller than the mean for all analysts. The quadrant (+, –), for example, contains results for analysts that exceeded the mean for sample X and that undershot the mean for sample Y. If the variation in results is dominated by random errors, then we expect the points to be distributed randomly in all four quadrants, with an equal number of points in each quadrant. Furthermore, as shown in Figure 14.19a, the points will cluster in a circular pattern whose center is the mean values for the two samples. When systematic errors are significantly larger than random errors, then the points occur primarily in the (+, +) and the (–, –) quadrants, forming an elliptical pattern around a line bisecting these quadrants at a 45^o angle, as seen in Figure 14.19b.

Figure 14.19 Typical two-sample plots when (a) random errors are significantly larger than systematic errors due to the analysts, and (b) when systematic errors due to the analysts are significantly larger than the random errors.

A visual inspection of a two-sample chart is an effective method for qualitatively evaluating the results of analysts and the capabilities of a proposed standard method. If random errors are insignificant, then the points fall on the 45^o line. The length of a perpendicular line from any point to the 45^o line, therefore, is proportional to the effect of random error on that analyst’s results. The distance from the intersection of the axes—corresponding to the mean values for samples X and Y—to the perpendicular projection of a point on the 45^o line is proportional to the analyst’s systematic error. Figure 14.20 illustrates these relationships. An ideal standard method has small random errors and small systematic errors due to the analysts, and has a compact clustering of points that is more circular than elliptical.

Figure 14.20 Relationship between the result for a single analyst (in blue) and the contribution of random error (red arrow) and the contribution from the analyst’s systematic error (green arrow).

We also can use the data in a two-sample chart to separate the total variation in the data, σ_tot, into contributions from random error, σ_rand, and systematic errors due to the analysts, σ_syst.⁹ Because an analyst’s systematic errors are present in his or her analysis of both samples, the difference, D, between the results

\[D_i = X_i - Y_i\]

is the result of random error. To estimate the total contribution from random error we use the standard deviation of these differences, s_D, for all analysts

where n is the number of analysts. The factor of 2 in the denominator of equation 14.18 is the result of using two values to determine D_i. The total, T, of each analyst’s results

\[T_i = X_i + Y_i\]

contains contributions from both random error and twice the analyst’s systematic error.

The standard deviation of the totals, s_T, provides an estimate for σ_tot.

Again, the factor of 2 in the denominator is the result of using two values to determine T_i.

If the systematic errors are significantly larger than the random errors, then s_T is larger than s_D, a hypothesis we can evaluate using a one-tailed F‑test

\[F=\dfrac{s_\textrm T^2}{s_\textrm D^2}\]

where the degrees of freedom for both the numerator and the denominator are n – 1. As shown in the following example, if s_T is significantly larger than s_D we can use equation 14.19 to separate σ_tot² into components representing random error and systematic error.

Figure 14.21 Two-sample plot for the data in Example 14.6. The number by each blue point indicates the analyst. The true values for each sample (see Example 14.7) are indicated by the red star.

If the true values for the two samples are known, we also can test for the presence of a systematic error in the method. If there are no systematic method errors, then the sum of the true values, μ_tot, for samples X and Y

\[\mu_\ce{tot} = \mu_\ce{X} + \mu_\ce{Y}\]

should fall within the confidence interval around T. We can use a two-tailed t-test of the following null and alternate hypotheses

\[H_0:\overline{T}= \mu_\ce{tot} \hspace{30px} H_\ce{A}: \overline{T}≠ \mu_\ce{tot}\]

to determine if there is evidence for a systematic error in the method. The test statistic, t_exp, is

\[t_{\textrm{exp}}=\dfrac{|\overline T-\,\mu_{\textrm{tot}}|\sqrt n}{s_\textrm T\sqrt2}\tag{14.21}\]

with n – 1 degrees of freedom. We include the √2 in the denominator because s_T (see equation 14.20) underestimates the standard deviation when comparing T to μ_tot.

Example 14.6 and Example 14.7 illustrate how we can use a pair of similar samples in a collaborative test of a new method. Ideally, a collaborative test involves several pairs of samples that span the range of analyte concentrations for which we plan to use the method. In doing so, we evaluate the method for constant sources of error and establish the expected relative standard deviation and bias for different levels of analyte.

14.3.2 Collaborative Testing and Analysis of Variance

In a two-sample collaborative test we ask each analyst to perform a single determination on each of two separate samples. After reducing the data to a set of differences, D, and a set of totals, T, each characterized by a mean and a standard deviation, we extract values for the random errors affecting precision and the systematic differences between then analysts. The calculations are relatively simple and straightforward.

An alternative approach for completing a collaborative test is to have each analyst perform several replicate determinations on a single, common sample. This approach generates a separate data set for each analyst, requiring a different statistical treatment to arrive at estimates for σ_rand and σ_syst.

There are several statistical methods for comparing three or more sets of data. The approach we consider in this section is an analysis of variance (ANOVA). In its simplest form, a one-way ANOVA allows us to explore the importance of a single variable—the identity of the analyst is one example—on the total variance. To evaluate the importance of this variable, we compare its variance to the variance explained by indeterminate sources of error.

We first introduced variance in Chapter 4 as one measure of a data set’s spread around its central tendency. In the context of an analysis of variance, it is useful for us to understand that variance is simply a ratio of two terms: a sum of squares for the differences between individual values and their mean, and the available degrees of freedom. For example, the variance, s², of a data set consisting of n measurements is

where X_i is the value of a single measurement and X is the mean. The ability to partition the variance into a sum of squares and the degrees of freedom greatly simplifies the calculations in a one-way ANOVA.

Let’s use a simple example to develop the rationale behind a one-way ANOVA calculation. The data in Table 14.6 are from four analysts, each asked to determine the purity of a single pharmaceutical preparation of sulfanilamide. Each column in Table 14.6 provides the results for an individual analyst. To help us keep track of this data, we will represent each result as X_ij, where i identifies the analyst and j indicates the replicate. For example, X_3,5 is the fifth replicate for the third analyst, or 94.24%.

The data in Table 14.6 show variability, both in the results obtained by each analyst and in the difference in the results between the analysts. There are two sources for this variability: indeterminate errors associated with the analytical procedure experienced equally by each analyst, and systematic or determinate errors introduced by individual analysts.

One way to view the data in Table 14.6 is to treat it as a single large sample, characterized by a global mean and a global variance

\[\overline{\overline X}=\dfrac{\sum_{i=1}^{h}\sum_{j=1}^{n_i}X_{ij}}{N}\tag{14.22}\]

where h is the total number of samples (in this case the number of analysts), n_i is the number of replicates for the ith sample (in this case the ith analyst), and N is the total number of data points (in this case 22). The global variance—which includes all sources of variability affecting the data—provides an estimate of the combined influence of indeterminate errors and systematic errors.

A second way to work with the data in Table 14.6 is to treat the results for each analyst separately. If we assume that each analyst experiences the same indeterminate errors, then the variance, s², for each analyst provides a separate estimate of σ_rand². To pool these individual variances, which we call the within-sample variance, s_w², we square the difference between each replicate and its corresponding mean, add them up, and divide by the degrees of freedom.

Equation 14.24 provides an estimate for σ_rand². To estimate the systematic errors, σ_syst², affecting the results in Table 14.6 we need to consider the differences between the analysts. The variance of the individual mean values about the global mean, which we call the between-sample variance, s_b², provides this estimate.

The between-sample variance includes contributions from both indeterminate errors and systematic errors

\[s_\ce{b}^2= σ_\ce{rand}^2 + \bar{n}σ_\ce{syst}^2\tag{14.26}\]

where n is the average number of replicates per analyst.

\[\bar n = \dfrac{\sum_{i=1}^{h}n_i}{h}\]

In a one-way ANOVA of the data in Table 14.6 we make the null hypothesis that there are no significant differences between the mean values for each analyst. The alternative hypothesis is that at least one of the means is significantly different. If the null hypothesis is true, then σ_syst² must be zero, and s_w² and s_b² should have similar values. If s_b² is significantly greater than s_w², then σ_syst² is greater than zero. In this case we must accept the alternative hypothesis that there is a significant difference between the means for the analysts. The test statistic is the F-ratio

\[F_{\textrm{exp}}=\dfrac{s_\ce{b}^2}{s_\ce{w}^2}\]

which is compared to the critical value F(α, h – 1, N – h). This is a one-tailed significance test because we are only interested in whether s_b² is significantly greater than s_w².

Both s_b² and s_w² are easy to calculate for small data sets. For larger data sets, calculating s_w² is tedious. We can simplify the calculations by taking advantage of the relationship between the sum-of-squares terms for the global variance (equation 14.23), the within-sample variance (equation 14.24), and the between-sample variance (equation 14.25). We can split the numerator of equation 14.23, which is the total sum-of-squares, SS_t, into two terms

\[SS_\ce{t} = SS_\ce{w} + SS_\ce{b}\]

where SS_w is the sum-of-squares for the within-sample variance and SS_b is the sum-of-squares for the between-sample variance. Calculating SS_t and SS_b gives SS_w by difference. Finally, dividing SS_w and SS_b by their respective degrees of freedom gives s_w² and s_b². Table 14.7 summarizes the equations for a one-way ANOVA calculation. Example 14.8 walks you through the calculations, using the data in Table 14.6. Section 14D provides instructions on using Excel and R to complete a one-way analysis of variance.

Having demonstrated that there is significant difference between the analysts, we can use a modified version of the t-test, known as Fisher’s least significant difference, to determine which analyst or analysts are responsible for the difference. The test statistic for comparing two mean values is the t-test given in equation 4.21 in Chapter 4, except we replace the pooled standard deviation, s_pool, by the square root of the within-sample variance from the analysis of variance.

\[t_{\textrm{exp}}=\dfrac{|\overline X_1-\overline X_2|}{\sqrt {s_\textrm w^2}\sqrt{\dfrac{1}{n_\textrm A}+\dfrac{1}{n_\textrm B}}}=\dfrac{|\overline X_1-\overline X_2|}{\sqrt{s_\textrm w^2}}\times \sqrt{\dfrac{n_\textrm An_\textrm B}{n_\textrm A+n_\textrm B}}\tag{14.27}\]

We compare t_exp to its critical value t(α, ν) using the same significance level as the ANOVA calculation. The degrees of freedom are the same as that for the within sample variance. Since we are interested in whether the larger of the two means is significantly greater than the other mean, the value of t(α, ν) is that for a one-tailed significance test.

We can extend an analysis of variance to systems involving more than a single variable. For example, we can use a two-way ANOVA to determine the effect on an analytical method of both the analyst and the instrumentation. The treatment of multivariate ANOVA is beyond the scope of this text, but is covered in several of the texts listed in this chapter’s additional resources.

14.3.3 What is a Reasonable Result for a Collaborative Study?

Collaborative testing provides us with a method for estimating the variability (or reproducibility) between analysts in different labs. If the variability is significant, we can determine what portion is due to indeterminate method errors (σ_rand²) and what portion is due to systematic differences between the analysts (σ_syst²). What we have left unanswered is the following important question: What is a reasonable value for a method’s reproducibility?

An analysis of nearly 10 000 collaborative studies suggests that a reasonable estimate for a method’s reproducibility is

\[R=2^{(1-0.5\log C)}\tag{14.28}\]

where R is the percent relative standard deviation for the results included in the collaborative study and C is the fractional amount of analyte in the sample on a weight-to-weight basis.¹⁰ Equation 14.28 appears to be independent of the type of analyte, the type of matrix, and the method of analysis. For example, when a sample in a collaborative study contains 1 microgram of analyte per gram of sample, C is 10^–6 and the estimated relative standard deviation is

\[R=2^{(1-0.5\log 10^{-6})}=16\%\]

Of course, equation 14.28 only estimates the expected relative standard. If the method’s relative standard deviation falls with a range of one-half to twice the estimated value, then it is acceptable for use by analysts in different laboratories. The percent relative standard deviation for a single analyst should be one-half to two-thirds of that for the variability between analysts.

Contributors

David Harvey (DePauw University)