ASTM C802-14(2022)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: C802 − 14 (Reapproved 2022)
Standard Practice for
Conducting an Interlaboratory Test Program to Determine
the Precision of Test Methods for Construction Materials
This standard is issued under the fixed designation C802; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope* C136 Test Method for Sieve Analysis of Fine and Coarse
Aggregates
1.1 This practice describes techniques for planning,
C311/C311M Test Methods for Sampling and Testing Fly
conducting, and analyzing the results of an interlaboratory
Ash or Natural Pozzolans for Use in Portland-Cement
study (ILS) with the objective of developing the precision
Concrete
statement of a test method. It is designed to be used in
C670 Practice for Preparing Precision and Bias Statements
conjunction with Practice C670. The methods used in this
for Test Methods for Construction Materials
standard are consistent with those in Practice E691.
C1067 Practice for Conducting a Ruggedness Evaluation or
1.2 This practice is not intended for use in programs whose
Screening Program for Test Methods for Construction
purpose is to develop a test method or to assess the relative
Materials
variability of two or more test methods. Refer to Practice
E105 Guide for Probability Sampling of Materials
C1067 for procedures to evaluate the ruggedness of a test
E177 Practice for Use of the Terms Precision and Bias in
method.
ASTM Test Methods
E178 Practice for Dealing With Outlying Observations
1.3 The system of units for this practice has not been
specified. Dimensional quantities in the practice are presented E456 Terminology Relating to Quality and Statistics
E691 Practice for Conducting an Interlaboratory Study to
only in examples of calculations.
Determine the Precision of a Test Method
1.4 This standard does not purport to address all of the
safety concerns, if any, associated with its use. It is the
3. Terminology
responsibility of the user of this standard to establish appro-
priate safety, health, and environmental practices and deter-
3.1 Definitions:
mine the applicability of regulatory limitations prior to use.
3.1.1 For definitions of general statistical terms, refer to
1.5 This international standard was developed in accor-
Terminology E456.
dance with internationally recognized principles on standard-
3.1.2 For definitions of terms associated with precision of
ization established in the Decision on Principles for the
test methods for construction materials, refer to Practice C670.
Development of International Standards, Guides and Recom-
mendations issued by the World Trade Organization Technical
4. Significance and Use
Barriers to Trade (TBT) Committee.
4.1 This practice provides requirements for planning and
2. Referenced Documents conducting an interlaboratory study to obtain data to develop
single-operator and multilaboratory precision statements for a
2.1 ASTM Standards:
test method. It includes methods to evaluate data consistency
C109/C109M Test Method for Compressive Strength of
before carrying out the calculations to develop the precision
Hydraulic Cement Mortars (Using 2-in. or [50 mm] Cube
statement. The procedures are compatible with those in Prac-
Specimens)
tice E691.
4.2 The ILS data obtained from this practice are intended
This practice is under the jurisdiction of ASTM Committee C09 on Concrete
for developing the precision values for writing single-operator
and ConcreteAggregates.This practice was developed jointly byASTM Committee
C01, C09, D04, and D18, and is endorsed by all four committees.
and multilaboratory precision statements in accordance with
Current edition approved Oct. 1, 2022. Published October 2022. Originally
Practice C670.
approved in 1974. Last previous edition approved in 2014 as C802 – 14. DOI:
10.1520/C0802-14R22.
4.3 Appendix X1 provides an example to illustrate the
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
calculations to obtain the precision values of the test method
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
from the ILS data.This may be used to check a user-developed
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website. electronic spreadsheet for carrying out the calculations.
*A Summary of Changes section appears at the end of this standard
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
C802 − 14 (2022)
4.4 Appendix X2 discusses the additional calculations re- There should be a reasonable degree of certainty that the
quired for an interlaboratory study of a test method that single-operator variances are the same in different laboratories,
includesmakingtestspecimensaspartoftheprocedure.Inthis and that troublesome interactions do not exist. These condi-
case, batch-to-batch variability needs to be considered. tions are investigated in the initial analysis of the data of an
interlaboratory study, and are discussed further in 10.4.
4.5 Appendix X3 discusses the use of analysis of variance
as an alternative approach to obtain the precision values from
5.5 Facilities and procedures for procurement, preparation,
the ILS data.
and distribution of samples or test specimens must be avail-
able.
5. General Requirements
5.6 Selection of samples or test specimens must be done by
5.1 Certain criteria need to be met before undertaking an
a randomization process, and one person who is familiar with
interlaboratory study to determine the precision of a test
randomization procedures needs to be responsible for seeing
method. If some conditions are not met or are met
that an appropriate randomization technique is used. Refer to
incompletely, the program will become more complicated to
Practice E105.
administer and require more work and expense, or may result
in impaired information. The requirements outlined in this
5.7 The precision of the test method should be evaluated on
section are intended to ensure that the test method is free of
different materials with a range of the characteristic being
technical difficulties to the greatest extent possible before an
measured that encompasses the typical use of the method in
expensive and time-consuming interlaboratory study is under-
practice. (See 7.1 and 7.2.)
taken.
5.8 Adequate numbers of participating laboratories,
5.2 The first requirement is the existence of a valid and
operators, and materials must be available. Requirements in
well-written test method that has been developed in one
these areas are specified in Sections 6 and 7.
laboratory and has been subjected to ruggedness evaluation of
5.9 The entire interlaboratory test program should be devel-
the testing procedure and conditions as described in Practice
oped from the beginning with the help and advice of persons
C1067. As a result of the screening procedure and some
familiar with statistical procedures and with the materials
experience with the test method in the sponsoring laboratory
involved. The ASTM International Interlaboratory Study Pro-
and one or two others, a written version of the test method has
been developed (but not necessarily published as a standard) gram can support subcommittees in the development of preci-
sion statements by assisting in the design of an interlaboratory
that describes the test procedure in terms that can be followed
by a competent operator in any properly equipped laboratory. study, distribution of specimens or samples, data analysis, and
preparation of a draft research report. Additional information
Critical conditions that affect the test results need to be
identifiedandtheproperandrealisticdegreeofcontrolofthose about theASTM ILS program can be obtained from theASTM
Website.
conditions have to be specified in the description of the test
procedure.
5.9.1 Itmaynotalwaysbepossibletoobtainpeoplewhoare
5.2.1 The tolerances established for various conditions in a familiar with the materials involved and who have a sufficient
test method provide reasonable ranges for these conditions and
knowledge of the proper statistical techniques and their proper
recognize that precise values with small tolerances may not be use. In this case, the subcommittee should obtain the services
achievable in practice. Variations in test results due to varia-
of a statistician who has experience in practical work with data
tions in such conditions contribute to the total variation, which from materials testing, and provide that person with an
determines the precision of the test method. If the resulting
opportunity for learning something about the particular mate-
variation is so great that uncertainties in average values rials and test method involved. Planners of an interlaboratory
obtained by the test method are unacceptably high, the test
study need to avoid the pitfall of assuming that the use of
method itself is at fault and it will need to be improved or statistical analysis software programs necessarily results in
replaced by a better one. An expensive and time-consuming
special expertise in manipulating the data or interpreting the
interlaboratory study is not recommended for such a test results.
method.
5.10 It is important to bear in mind that estimates of the
5.2.2 Apparatus required for performing the test must be
precision of a test method are always based on a particular set
defined clearly and must be available or able to be produced. If
of data obtained at a particular time and precision values need
alternative apparatus is permitted, criteria need to be provided
to be kept up-to-date. As materials, apparatus, and conditions
on the performance requirements of the apparatus, such as by
change, and operators change or gain more experience, the
specifying acceptable limits of measurements on standard
characteristic precision of the results obtained may change,
reference materials.
especially if the test method is new. In some cases, it may be
5.3 Personnel in laboratories participating in the ILS should
desirable to conduct more tests at a later date (though not
have adequate experience with routine laboratory procedures
necessarily a repetition of the complete interlaboratory study)
so that they are competent to run the test. The importance of
in order to provide a check on estimates previously obtained
this requirement will vary with the complexity of the method
and either verify them or introduce revisions. When a subcom-
and the degree to which it departs from familiar procedures.
mittee revises a test method, it should consider whether the
5.4 It is helpful to have preliminary knowledge about how proposed changes might affect the precision of the method. If
changes in materials and conditions affect the test results. there is a possibility that precision will be affected, limited
C802 − 14 (2022)
interlaboratory testing is recommended to evaluate whether the 7.1.3 The difficulty and expense involved in obtaining,
existing precision statement is still applicable or if a new ILS processing, and distributing samples or specimens;
needs to be organized to better reflect the precision of the
7.1.4 The difficulty of, length of time required for, and
revised method. expense of performing the tests; and
7.1.5 The uncertainty of prior information on any of these
6. Laboratories
points. For example, if it is already known that the precision is
6.1 Obtaining participating laboratories for an interlabora- relatively constant or proportional to the average level over the
torystudyisoftenoneofthemostdifficultproblemsconnected
range of values of interest, a smaller number of materials will
with the process. The number of laboratories available is be needed than if it is known that the precision changes
seldom as extensive as one would like, and if the test method
erratically at different levels. A preliminary pilot or screening
is new, complicated, or expensive and time-consuming to run,
program may help to settle some of these questions, and may
the problem is further complicated.
often result in the saving of considerable time and expense in
the full interlaboratory study (1).
6.2 For the purposes of programs using this practice, it is
recommended that at least ten competent laboratories be
7.2 In general, at least three materials or three different
included (1, 2). In cases where it is impossible to obtain ten
average values of the measuredtest characteristicisconsidered
laboratories,theeffectofanincreasednumbermaybeobtained
acceptable.Thematerialsneedtobeselectedtoobtainasbroad
by repeating the program with the same group of laboratories
a range of the test characteristic as is practicable. If the test
sixmonthslater.Ifthisprocedureisfollowed,itisnecessaryto
method is used to determine properties that are used for
be sure that the same materials are used, and that their
acceptancetestinginaspecification,itisparticularlyimportant
characteristics have not changed in the interim. This approach,
that materials be included in the ILS whose properties are near
however, may not provide a proper measure of the between-
the specification limits.
laboratory component of variance, unless different operators or
7.3 Specimen Distribution—The ILS is based on the as-
equipment, or both, are used for the repeat testing. In any case,
sumption that all tests are performed on specimens that are as
six is the absolute minimum number of laboratories for
similar as is possible. Generally, two approaches are used for
evaluating the precision of a test method. This means that at
makinganddistributingthespecimensormaterialsfortheILS.
least seven to eight laboratories should be in the ILS study in
7.3.1 For a test method that does not involve production of
case problems are encountered with the data provided by a
the test specimens as part of the method, specimens are
participating laboratory.
produced at one location from a homogenous sample and then
6.3 In general, it is recommended that any laboratory that is
distributed to the participating laboratories. The specimens
considered qualified to run the test in routine testing situations
need to be assigned to the participating laboratories on a
should be permitted and encouraged to participate. “Qualified”
randombasis.Ifthecharacteristictobemeasuredchangeswith
implies proper laboratory facilities and testing equipment,
age, specific instructions on test age need to be provided.
competent operators familiar with routine laboratory
7.3.2 For a test method that involves fabrication of test
techniques, a history of reliable testing work, and sufficient
specimens as part of the method, the raw materials for making
time and interest to do a good job. It does not mean, however,
the test specimens are shipped to the participating laboratories.
thatonlyaselectgroupoflaboratoriesthatareconsideredtobe
Inthiscase,samplesoftheconstituentmaterialsaretakenfrom
those best qualified for the interlaboratory study should be
homogenous blends of the materials. The samples are selected
picked. Precision estimates for inclusion in a test method must
on a random basis for shipment to the participating laborato-
be obtained under conditions and through the efforts of
ries. Facilities are needed that have the proper equipment for
laboratories and personnel that are representative of the situa-
blending the materials.
tions in which the test method will be used in practice (2).Ifa
7.3.3 In some cases, it is not possible to distribute materials
laboratory satisfies all the other requirements, but its personnel
to participating laboratory because of the nature of the material
has had insufficient experience with the method, the operators
or effects of transportation or age. This may require operators
in that laboratory should be given an opportunity to familiarize
from participating laboratories to convene at one location to
themselves with the method and to practice its application
test the materials. This procedure is used commonly in devel-
before the interlaboratory study starts.
oping precision statements for fresh concrete test methods.
7. Materials
8. Estimates of Precision
7.1 Number—The number of materials to be included in an
8.1 In accordance with Practice C670, the procedure de-
interlaboratory study will depend on the following:
scribed in this practice is designed to provide data to develop
7.1.1 The range of the values of the property that may be
two basic estimates of the precision of a test method: (a)
measured in practice and how the precision varies over that
single-operatorprecision,and(b)multilaboratoryprecision (1).
range;
(See Note 1.)
7.1.2 The types of materials to which the test method is to
be applied;
8.2 Single-operator precision provides estimates of the
inherent variability of the test method and the maximum
difference that may be expected between replicate measure-
The boldface numbers in parentheses refer to the list of references at the end of
this practice. ments made on the same material in the same laboratory by the
C802 − 14 (2022)
same operator using the same apparatus within a time span of determinations to be obtained and reported. If a test result is
a few days. The words “may be expected” mean that there is 5 defined, either in the test method or in the instructions to
% likelihood that the difference will exceed the stated maxi- laboratories participating in an interlaboratory test program, as
mum difference, even if testing conforms to the test method. In the average of a particular number of determinations, the
Practice C670, the maximum acceptable difference is referred individual determinations shall always be reported, in addition
to as the “difference limit (d2s)” or “difference limit (d2s%)” if to the averages.
the coefficient of variation is the appropriate measure of 9.2.2 Rounding of Data:
precision. 9.2.2.1 Generally, laboratories need to report all figures
obtainedinmakingthemeasurements,ratherthanroundingthe
8.3 Multilaboratory precision provides estimates of the
results before recording them. In some cases, this may result in
variability among laboratories and the maximum difference
recording of more digits than is customary or even more than
thatmaybeexpectedbetweenmeasurementsmadeonthesame
the test method calls for in the section on Reporting. This is
material in two different laboratories.
necessary because the variation from which information about
8.4 Ifestimatesofprecisionduetootherfactorsarerequired
the precision of the test method comes is contained in the least
forthetestmethod,theILSneedstobeplannedtoprovidedata
significant digits, which are often discarded in reporting the
to develop the appropriate statistics for the systems of causes
results of routine testing (3). For example, Test Method C136
being considered and the appropriate combination of modifiers
calls for reporting the percentage retained on a sieve to the
given in Practice E177 should be used to describe those
nearest whole number. This is adequate for the usual reporting
statistics.Theadviceofastatisticalconsultantisrecommended
purposes, but for purposes of determining precision, at least
for these cases.
one decimal place is needed. It is better to require the reporting
NOTE 1—Appendix X2, for example, explains how to analyze ILS data
of too many decimal places than too few, because a decision
for developing the single-operator, multi-batch precision of a test method
about rounding all data can be made when the analysis is done.
that involves making the test specimens as part of the procedure.Another
If too few places are reported, however, valuable information
example is developing the single-operator, multi-day precision, which
would involve the additional variability due to testing on different days.
may be irretrievably lost, and the result might well be the
impairment of the entire program.
9. Collection of Data
9.2.2.2 If a test determination is the result of a calculation
9.1 In order to minimize the problems concerned with
based on two or more measured quantities, the basic measure-
analysis of data, a definite form and instructions for obtaining
ments should be used in the calculations without any rounding.
and recording the data have to be developed and distributed to
The planners of the interlaboratory program will then have to
all participating laboratories.
determine how many places need to be reported in order to
retain the essential information for determining variability.
9.2 Directions to Laboratories—The directions to the labo-
Sometimes it is advisable to ask the laboratories to report the
ratories should deal mainly with reporting of data. No special
basic quantities measured instead of, or in addition to, the final
instructions for performing the tests that differ from those
calculated result. This enables the final result to be checked, or
given in the test method should be included. The laboratories
changesindecisionsaboutthetestresultstobemade,whenthe
must be instructed to conduct tests and report results exactly as
data are analyzed. An example would be a strength test for
specified in the test method, with the one exception as noted in
which the measured specimen dimensions along with the
9.2.2. Often data are disseminated in digital form, but labora-
ultimate load should be reported so that the reported strength
tories need to maintain hard-copies of their data to provide
can be verified.
documentation in the event of digital data corruption.
9.2.1 Averaging Test Determinations—Laboratories should 9.3 The Data Sheet—This practice is based on the following
particularly be cautioned against practices such as running a assumptions: p laboratories each have made n replicate deter-
number of tests and selecting the “best” ones or reporting the minations on each of q materials (4). Table 1 and Table 2 are
average of several determinations, except as such averaging is examples of data sheets for an individual laboratory and for a
specified in the test method. For example, Test Method summary of data for the entire ILS program with: p = ten
C109/C109M specifies three or more test specimens, and laboratories, n = four replicate determinations, and q = five
requires that the strength of all acceptable test specimens made materials.These data sheets suggest the format to be used if an
from the same batch and tested at the same period shall be
individual determination constitutes a test result. If individual
averaged and reported. In this case, the directions for the determinations are averaged or otherwise subjected to calcula-
interlaboratory study must specify the number of individual
tion to produce a test result, the format of the individual
TABLE 1 Data Sheet for an Interlaboratory Test Program for an ASTM Test Method
Laboratory: XX
Material
Replicate
AB C D E
a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
C802 − 14 (2022)
TABLE 2 Summary Data Sheet for an Interlaboratory Test Program for an ASTM Test Method
Material
Laboratory Replicate
AB C D E
1 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
2 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
3 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
4 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
5 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
6 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
7 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
8 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
9 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
10 a _______ _______ _______ _______ _______
b _______ _______ _______ _______ _______
c _______ _______ _______ _______ _______
d _______ _______ _______ _______ _______
laboratory data sheet may be altered or a secondary sheet estimates of single-operator precision but has no effect on
provided to permit recording the fundamental measurements
between-laboratory precision (2). It is recommended that, for
and the test results. 10 to 15 participating laboratories, at least three replicates are
required. If it is not possible to obtain 10 participating
9.4 Number of Replicates—Even if the test method calls for
laboratories, the number of replicates, n, should be at least
a single determination as a test result, replicate determinations
(30/p) + 1. If 30 is not a multiple of p, 30/p is rounded to the
are required in the ILS in order to obtain information for
next higher integer. For more than 15 laboratories, the number
calculating the single-operator precision.
ofreplicatesmaybereducedtotwo.Thiswillgiveanadequate
9.4.1 The number of replicate determinations to be made on
estimate of single-operator precision, but information about
eachmaterialineachlaboratorydependslargelyonthenumber
multilaboratory precision is not as good as desired with fewer
of laboratories participating, on the homogeneity of the
than 10 laboratories.
material, and on the expense, difficulty, and time involved in
increasing the number of determinations. In order to obtain the 9.4.2 The variation among replicate determinations is sup-
posed to be representative of the irreducible error variance
necessary information to write a meaningful precision
statement, it is often necessary to use more replicates in the characteristicofthetestmethod.Insomecases,itispossibleto
interlaboratory study than is required for routine use of the test take supposedly replicate measurements in such a manner that
method. An increase in the number of replicates improves the there is little or no opportunity for chance variation; and the
C802 − 14 (2022)
measurements are in effect simply repetitions of the same on four determinations. If the number of missing results
measurement. For example, in making a chemical analysis by exceeds3%ofthe total, some of the tests should be repeated
atomic absorption or some other kind of automatic measuring in order to obtain proper measurements for the missing values.
device, laboratories have been known to take three readings of Missing values handled in this way must be individual values
the meter on the same sample in quick succession. The three distributed throughout the mass of data, and are not to be
readings so taken were almost identical, but were still reported concentrated as a group in one laboratory-material cell (see
as replicate readings. In cases such as this, three separate Note 2). If the latter occurs, the laboratory should provide
readings with different portions of the sample or with separate another group of measurements on the material in question.
specimens should be obtained, with the same operator and
NOTE 2—A cell refers to the group of replicate test determinations for
a particular combination of laboratory and material. Appendix X3
apparatus to provide meaningful replicate measurements.
describes an analysis-of-variance procedure that can be used to analyze
9.5 Outliers—Section 10.4 describes a procedure for iden-
unbalanced sets of data. The advice of a statistical consultant should be
tifying test determinations that have unexpected variations obtained if such procedures are used.
fromthoseobtainedbyotherparticipantsintheILS.Ingeneral,
the practice of discarding individual test determinations, which 10. Analysis of Data
appear to differ by suspiciously large amounts from the others,
10.1 An ILS is a type of experiment design known as a
is to be avoided unless there is clear evidence that there was
nesteddesignorahierarchaldesign (6).Thegeneralpurposeof
some physical reason to consider the determination faulty.
a nested design is to identify and quantify the sources of
Discarding results with unexpected deviations but without a
variation in a process. In the case of the ILS, the objective is to
proper basis or an assignable cause for the deviations may
quantify the single-operator and between-laboratory compo-
resultinunrealisticprecisionvaluesthatmaynotberelevantto
nents of variance. Fig. 1 is a diagram of a single-stage nested
the test method. On the other hand, retaining invalid results
design that is representative of the basic ILS described in this
with unexpectedly large variations may result in precision
practice. The laboratories participating in the ILS are the first
values that tolerate less careful testing. Laboratories must be
stage representing the factor “laboratory.” For each laboratory,
instructed to report all results in their proper place and include
there are n replicate determinations. The replicate determina-
notes describing the conditions surrounding those results that
tions are considered to be “nested” within the factor “labora-
are suspected of being faulty. Sometimes if a test really went
tory.”
wrong, a laboratory should discard the results and repeat the
test. Tests are not to be repeated, however, just because the
10.2 The procedure described herein is simplified, and
results don’t look good. Further guidance on dealing with statistical terms are avoided to the greatest extent possible in
outliers is given in Practice E178 and in Refs (2, 5).
order to make the practice usable by persons with little
statistical background. This exposes the practice to the danger
9.6 Missing Data—The method of data analysis used in this
that,althoughthetechniquerecommendediswidelyapplicable
practices assumes a balanced set of data, which means that
to many situations using many kinds of data, it may be used
each laboratory provides the required number of determina-
mechanically in situations for which it is not applicable. For
tions for all materials. Sometimes individual determinations
this reason, it is recommended to seek the advice of a person
are missing from the summary because they were discarded,
who is familiar with the statistical procedures before undertak-
failed to be supplied by a laboratory, or for other reasons. In
ing analysis of ILS data by this or any other published
general, if the number of missing data items from all labora-
procedure. An example of the procedure is given in Appendix
tories constitutes no more than about 3 % of the total number
X1. For further description of the method, see Ref (1). The
of items, the analysis may be conducted as though the missing
items were present. For example, if one out of four required
replicate determinations on a given material from laboratory, i,
out of 10 laboratories is missing, the three reported determi-
nations should be added and divided by three to obtain the
¯
average, X. The single-operator variance, s , should be
i ri
calculated using three for the number of determinations. From
then on, both values should be used as though they were based
FIG. 1 Diagram of a Single-Stage Nested Experiment Design
C802 − 14 (2022)
A
TABLE 3 Single-Operator and Between-Laboratory Analysis for Material A
Data (Replicates)
Single-Operator
Average,
Laboratory
¯
X
Variance, s
ab c d iA
riA
¯ 2
1 ———— X
s
1A
r1A
¯
2 ———— X
2A s
r2A
¯ 2
3 ———— X
3A s
r3A
¯ 2
4 ———— X
s
4A
r4A
¯ 2
5 ———— X
5A s
r5A
¯ 2
6 ———— X
s
6A
r6A
¯ 2
7 ———— X
7A s
r7A
¯ 2
8 ———— X
s
8A
r8A
¯ 2
9 ———— X
9A s
r9A
¯ 2
10 ———— X
s
10A
r10A
A
p = 10 laboratories
n = 4 replicate test determinations on each material in each laboratory
¯
ΣX
iA
¯
Overall average X 5
A
p
Σs
riA
Pooled single-operator variance = s 5
rA
p
¯ ¯
ΣsX 2 X d
2 iA A
Variance of laboratory averages = s ¯ 5
X
A p21
s
2 rA
Between-laboratory component of variance = s 5s 2
¯
LA
X
A
n
procedure does not require sophisticated software and can be laboratory. The lower case letters (a, b, c, d) in Table 3
implemented using an electronic spreadsheet. represent the replicate test determinations. The subscript i is
used to designate a particular laboratory in the analysis and
10.3 Single-OperatorandBetween-LaboratoryComponents
goes from 1 to p, where p is the total number of laboratories.
of Variance for Each Material—Before starting the analysis,
The subscript j is used to designate the different materials, and
plot the data. This can be done by making a scatter plot of the
goes from 1 to q, where q is the total number of materials. As
test determinations for each laboratory. A separate plot can be
shown in Table 1 and Table 2, the different materials are
made for each material, or all data can be shown in one plot.
identified with capital letters A, B, C, and so forth.
These plots will reveal any potential data inconsistencies that
¯
10.3.1 Single-operator analysis—The averages, X , and
will be investigated further in accordance with 10.4. The first
iA
variances, S , in the last two columns of Table 3 are the
stepintheanalysisistoobtainestimatesofsingle-operatorand
riA
single-operator averages and variances for the given material
between-laboratory components of variance for each material.
(in this example, it is Material A). These quantities are
This may be done by setting up the data as shown in Table 3
calculated from the n replicate test determinations within each
and using the equations presented in this section. Table 3 is set
of the p laboratories as follows:
up as an example using MaterialAfor tests in ten laboratories
(p = 10) with four replicate determinations per laboratory (n =
single test determination g by labo-
x
gij =
4)tocorrespondwiththeexamplesummarydatasheetinTable
ratory i for material j
2. Each row of data represents a particular laboratory-material
combination and often called a cell. Tables similar to Table 3
Σx
gij
¯ average of n replicate test determi-
X 5 (1)
would be used for each material in the study. In the equations
ij =
n
nations for laboratory i on material j
that follow, the subscript g is used to designate a single test
determination for a particular material in one laboratory and
goes from 1 to n, where n is the number of replicates for each ¯ single-operator variance of replicate
Σsx 2 X d
gij ij
s 5 (2)
= determinations for laboratory i on
rij
n2 1
material j
A statistical software package called Dataplot® is available from the National
InstituteofStandardsandTechnologythatwillperformtheplottingandcalculations
10.3.2 Between-laboratory analysis—From the single-
described in this practice. The program can be downloaded from homepage:
operator averages, Eq 1, and variances, Eq 2, for each
http://www.nist.gov/itl/sed/dataplot.cfm. Instructions on how to analyze ILS data
laboratory, the following quantities are calculated for the given
can be found at this site: http://www.itl.nist.gov/div898/software/dataplot/refman1/
auxillar/e691.htm. material: (1) the pooled single-operator variance, (2) the
C802 − 14 (2022)
overallaverage,(3)thevarianceoflaboratoryaverages,and(4) practice is the same as inPracticeE691.Two statisticsareused
the between-laboratory component of variance. These values to evaluate data consistency: (1) the h-value and (2) the
are entered at the bottom of Table 3 and are calculated as k-value.
follows (Note 3):
10.4.1 Check Laboratory Averages—The h-value is used to
check whether the average value for a laboratory is consistent
Σs
rij pooled single-operator vari-
with the overall average of the other laboratories for a given
s 5 (3)
=
rj
p
ance for material j (Note 4)
material. The h-value is calculated for each laboratory and
material as follows:
¯
¯ ¯
ΣX
ij overall average for all labora- X 2 X
¯
ij j
X 5 (4)
=
j
h 5 (7)
p tories for material j
ij
¯
s
Xj
where:
¯ ¯
ΣsX 2 X d ¯
2 ij j variance of laboratory aver- X = averageofresultsforlaboratory iandmaterial j(Eq1),
ij
s ¯ 5 (5) =
X
j
p2 1 ages for material j
¯
X = overall average of results for material j (Eq 4), and
j
s = standard deviation of laboratory averages for material
¯
Xj
j, which is the square root of Eq 5.
between-laboratory compo-
nent of variance for material
s
2 rj
2 j. If the calculated values is 10.4.2 Check Laboratory Dispersion—The k-value for each
s 5 s ¯ 2 (6)
Lj =
X
j n
negative, the between-
laboratory is used to check the consistency of the single-
laboratory component of vari-
operator variability for a given material. The k-value is
ance is taken as zero.
calculated for each laboratory and material as follows:
NOTE 3—Appendix X1 includes an example showing how these
s
calculations are made for each material.
rij
k 5 (8)
ij
NOTE 4—The method of pooling variances used here applies only if the
s
rj
individual variances are based on the same number of replicate tests. In
general, a pooled estimate of a variance is not obtained by averaging where:
individual variances if the number of replicate determinations is not the
s = single-operator standard deviation of replicate determi-
rij
same for all laboratories. Refer to a textbook on basic principles of
nations for laboratory i and material j, which is the
statisticsforthemethodtopoolvariancesifthenumberofreplicatesisnot
square root of Eq 2, and
constant.
s = pooled single-operator standard deviation for material
rj
10.4 Data Consistency—Beforecontinuingwiththeremain-
j, which is the square root of Eq 3.
ing analysis of the ILS data to determine the precision of the
test method, it is necessary to check each laboratory’s data for 10.4.3 Critical h- and k-values—The calculated h- and
consistency in terms of the average and the dispersion of the k-values are compared with the critical values shown in Table
results. If data from one laboratory are not consistent with data 4, which is extracted from a larger table in Practice E691. The
from the other laboratories, it may be necessary to eliminate second column gives the critical h-value, which depends only
that laboratory’s data before completing the analysis. Incon- on the number of laboratories. The subsequent columns give
sistent data may inflate the calculated precision values and the critical k-values, which depend on the number of labora-
thereby encourage laboratories to tolerate less careful testing. tories and the number of replicate test determinations. The
The approach for checking data consistency used in this h-values can be positive or negative, while the k-values are
A
TABLE 4 Critical Values of h and k at the 0.5 % Significance Level
Critical values of k
p Critical
Number of replicates, n
No. of value of
Labs h (±)
3 1.15 1.72 1.67 1.61 1.56 1.52
4 1.49 1.95 1.82 1.73 1.66 1.60
5 1.74 2.11 1.92 1.79 1.71 1.65
6 1.92 2.22 1.98 1.84 1.75 1.68
7 2.05 2.30 2.03 1.87 1.77 1.70
8 2.15 2.36 2.06 1.90 1.79 1.72
9 2.23 2.41 2.09 1.92 1.81 1.73
10 2.29 2.45 2.11 1.93 1.82 1.74
11 2.34 2.49 2.13 1.94 1.83 1.75
12 2.38 2.51 2.14 1.96 1.84 1.76
13 2.41 2.54 2.15 1.96 1.84 1.76
14 2.44 2.56 2.16 1.97 1.85 1.77
15 2.47 2.57 2.17 1.98 1.86 1.77
16 2.49 2.59 2.18 1.98 1.86 1.77
17 2.51 2.60 2.19 1.99 1.86 1.78
18 2.53 2.61 2.20 1.99 1.87 1.78
19 2.54 2.62 2.20 2.00 1.87 1.78
20 2.56 2.63 2.21 2.00 1.87 1.79
A
The above critical values for the h and k consistency statistics were calculated from Student’s t and the F-ratio as described in Practice E691.
C802 − 14 (2022)
always positive. The critical values in Table 4 are the 0.5 % a small k-value is not usually as troublesome as that of a large
significance level. According to Practice E691, this signifi- k-value. If one laboratory, however, performs its tests in such a
cance level was chosen on the basis of judgment and experi-
way that the normal causes of variation are not permitted to
ence so that not too many nor too few laboratories are flagged
occur, there may be an unrealistically low single-operator
for further investigation. Appendix X1 of Practice E691
standard deviation. Small k-values may indicate an insensitive
provides the basis for the critical values of h and k. Refer to
measurement scale or other measurement problems. If all the
Practice E691 for applicable values of h and k if more than 20
k-values are erratic, the test method is in trouble. Efforts to
laboratories or more than 6 replicate determinations are in-
develop precision statements from the data should be sus-
volved in the ILS.
pended and further study of the test method should be
10.4.4 Summary of h- and k-values—The h- and k-values
undertaken to determine the causes for such erratic behavior.
calculated for each laboratory and each material are assembled
Theadviceofastatisticalconsultantshouldbeobtainedifthere
in a table.Values that exceed the critical values and values that
is doubt about eliminating a laboratory with a high or low
approach the critical values should be highlighted. The h- and
k-value.
k-values should also be plotted as bar graphs grouped in two
10.4.6 Plots by Material—If a plot by laboratory shows
ways: (1) by laboratories and (2) by materials. The critical
several h-or k-values near the critical values, look at the
values should be drawn on the plots. The plots of h and k and
corresponding plot by material to see how that laboratory
the marked tables give a picture of the overall character of the
differs from the rest for a given material. Often an h-value that
variability of the test method as well as singling out particular
seemsstrongintheplotbylaboratory,becauseofitsrelationto
laboratories that should be investigated.
thevaluesfortheothermaterials,willturnouttobereasonably
10.4.5 Plots by Laboratories—Examples of plots of h and k
consistent with the other laboratories for the same material. On
by laboratories are shown in Appendix X1 based on the
the other hand, the h-or k-value for the one laboratory may be
illustrative data. For each laboratory, the materials are grouped
revealed as strongly different from the values for the other
in increasing order of the overall average property value. The
laboratories in the plot by material. If so, this behavior should
following guidelines can be used to evaluate differences
be investigated.
between laboratories.
10.4.7 Interactions—A common problem with test results
10.4.5.1 h-Plot—The h-plot indicates how the laboratory
obtained from an interlaboratory study is the presence of
average property values for each material compare with the
interactions among laboratories and materials. This means that
overall average for that material. There are several general
the pattern of the results obtained on the material by one
patterns in these plots. In one, all laboratories have both
laboratory differs from the pattern obtained by the other
positive and negative h-values among the materials. In the
laboratories. In extreme cases, different laboratories may even
second, individual laboratories tend to have either positive or
fail to rate materials in the same order based on the measured
negative h-values for all materials, and the number of labora-
average properties. The accepted statistical technique for
tories with negative values is approximately the same as the
finding significant interactions is an analysis of variance of the
number of laboratories with positive values. Neither of these
total ILS data (all materials included). A reasonably reliable
patternsisunusualorrequiresinvestigation,althoughtheymay
method for checking to see if troublesome interactions may
tellsomethingaboutthenature
...