ASTM E691-23

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E691 − 23 An American National Standard
Standard Practice for
Conducting an Interlaboratory Study to Determine the
Precision of a Test Method
This standard is issued under the fixed designation E691; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
This standard has been approved for use by agencies of the U.S. Department of Defense.
1. Scope 1.6 This international standard was developed in accor-
dance with internationally recognized principles on standard-
1.1 This practice describes the techniques for planning,
ization established in the Decision on Principles for the
conducting, analyzing, and treating the results of an interlabo-
Development of International Standards, Guides and Recom-
ratory study (ILS) of a test method. The statistical techniques
mendations issued by the World Trade Organization Technical
described in this practice provide adequate information for
Barriers to Trade (TBT) Committee.
formulating the precision statement of a test method.
1.2 This practice does not concern itself with the develop- 2. Referenced Documents
ment of test methods but rather with gathering the information 2
2.1 ASTM Standards:
needed for a test method precision statement after the devel-
E29 Practice for Using Significant Digits in Test Data to
opment stage has been successfully completed. The data
Determine Conformance with Specifications
obtained in the interlaboratory study may indicate, however,
E177 Practice for Use of the Terms Precision and Bias in
that further effort is needed to improve the test method.
ASTM Test Methods
E456 Terminology Relating to Quality and Statistics
1.3 Since the primary purpose of this practice is the devel-
opment of the information needed for a precision statement, the E1169 Practice for Conducting Ruggedness Tests
E1402 Guide for Sampling Design
experimental design in this practice may not be optimum for
evaluating materials, apparatus, or individual laboratories. E2282 Guide for Defining the Test Result of a Test Method
1.4 Field of Application—This practice is concerned exclu-
3. Terminology
sively with test methods which yield a single numerical figure
3.1 Definitions—Unless otherwise noted in this standard, all
as the test result, although the single figure may be the outcome
terms relating to quality and statistics are defined in E456.
of a calculation from a set of measurements.
3.1.1 accuracy, n—the closeness of agreement between a
1.4.1 This practice does not cover methods in which the
test result and an accepted reference value. E177
measurement is a categorization; however, for many practical
3.1.2 bias, n—the difference between the expectation of the
purposes categorical outcomes can be scored, such as zero-one
test results and an accepted reference value. E177
scoring for binary measurements or as integers, ranks for
example, for well-ordered categories and then the test result
3.1.3 interlaboratory study, (ILS) in ASTM, n—a designed
can be defined as an average, or other summary statistic, of
procedure for obtaining a precision statement for a test method,
several individual scores.
involving multiple laboratories, each generating replicate test
results on one or more materials.
1.5 This standard may involve hazardous materials,
operations, and equipment. This standard does not purport to
3.1.4 observation, n—the process of obtaining information
address all of the safety concerns, if any, associated with its
regarding the presence or absence of an attribute of a test
use. It is the responsibility of the user of this standard to
specimen, or of making a reading on a characteristic or
establish appropriate safety, health, and environmental prac-
dimension of a test specimen. E2282
tices and determine the applicability of regulatory limitations
3.1.5 precision, n—the closeness of agreements between
prior to use.
independent test results obtained under stipulated conditions.
E177
This practice is under the jurisdiction of ASTM Committee E11 on Quality and
Statistics and is the direct responsibility of Subcommittee E11.20 on Test Method
Evaluation and Quality Control. For referenced ASTM standards, visit the ASTM website, www.astm.org, or
Current edition approved April 1, 2023. Published April 2023. Originally contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
approved in 1979. Last previous edition approved in 2022 as E691 – 22. DOI: Standards volume information, refer to the standard’s Document Summary page on
10.1520/E0691-23. the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E691 − 23
3.1.6 repeatability, n—precision of test results from tests laboratory’s cell average compares with the average of the
conducted within the shortest practical time period on identical other laboratories for a particular material (see A1.2.2).
material by the same test method in a single laboratory with all
3.2.3 between-laboratory standard deviation, s , n—the
L
known sources of variability conditions controlled at the same
sample standard deviation attributable to differences of test
levels (see repeatability conditions). E177
result means among laboratories.
3.1.7 repeatability conditions, n—conditions where inde-
3.2.4 between-laboratory variance, s , n—the sample vari-
L
pendent test results are obtained with the same method on
ance component attributable to differences of test result means
identical test items in the same laboratory by the same operator
among laboratories.
using the same equipment within short intervals of time. E177
3.2.4.1 Discussion—This statistic is estimated indirectly
3.1.8 repeatability limit (r), n—the value below which the
from the variance of cell averages and the repeatability
absolute difference between two individual test results obtained
variance. In situations where there is good agreement among
under repeatability conditions may be expected to occur with a
laboratories the estimate of this variance component may be
probability of approximately 0.95 (95 %). E177
close to zero or be negative. In the latter case, the estimate is
set to zero. (See Note 2 and A1.1.2.)
3.1.9 repeatability standard deviation, (s ), n—the standard
r
3.2.5 cell, n—the intersection of a row and column in a
deviation of test result obtained under repeatability conditions.
two-way classification table, in which the rows represent the
E177
laboratories and the columns represent the materials.
3.1.10 reproducibility, n—precision of test results from tests
3.2.5.1 Discussion—The table holds the test results from an
conducted on identical material by the same test method in
interlaboratory study, and each cell contains the test results
different laboratories (see reproducibility conditions). E177
from a particular laboratory on a particular material (see
3.1.11 reproducibility conditions, n—conditions where test
Section 7 and Table 1).
results are obtained with the same method on identical test
3.2.6 cell average, x¯, n—the average of the test results in a
items in different laboratories with different operators using
particular cell.
different equipment. E177
3.2.7 cell deviation, d, n—the cell average minus the aver-
3.1.12 reproducibility limit (R), n—the value below which
age of the cell averages.
the absolute difference between two test results obtained under
3.2.8 cell standard deviation, s, n—the standard deviation of
reproducibility conditions may be expected to occur with a
the test results in a particular cell.
probability of approximately 0.95 (95 %). E177
3.2.9 repeatability variance, s , n—the sample variance of
r
3.1.13 reproducibility standard deviation (s ), n—the stan-
R
test results obtained under repeatability conditions.
dard deviation of test results obtained under reproducibility
conditions. E177
3.2.9.1 Discussion—This statistic is estimated for a material
as the pooled within-laboratory variances over all of the
3.1.14 ruggedness test, n—a planned experiment in which
laboratories in the ILS.
environmental factors or test conditions are deliberately varied
3.2.10 reproducibility variance, s , n—the sample variance
R
in order to evaluate the effects of such variation. E1169
of test results obtained under reproducibility conditions.
3.1.15 test determination, n—the value of a characteristic or
3.2.10.1 Discussion—This statistic is estimated as the sum
dimension of a single test specimen derived from one or more
of the two variance components due to between-laboratories,
observed values. E2282
2 2
s , and within-laboratories, s .
L r
3.1.16 test method, n—a definitive procedure that produces
3.2.11 standard deviation of the cell averages, s , n—the
x¯
a test result. E2282
standard deviation of the cell averages for a particular material.
3.1.17 test observation, n—see observation. E2282
3.2.12 variance of the cell averages, s , n—the sample
x¯
3.1.18 test result, n—the value of a characteristic obtained
variance of the cell averages for a particular material.
by carrying out a specified test method. E2282
3.2.13 within-laboratory consistency statistic, k, n—the ra-
3.1.19 test specimen, n—the portion of a test unit needed to
tio of the cell standard deviation to the repeatability standard
obtain a single test determination. E2282
deviation.
3.2.13.1 Discussion—This statistic is an indicator of how
3.1.20 test unit, n—the total quantity of material (containing
one laboratory’s cell standard deviation under repeatability
one or more test specimens) needed to obtain a test result as
conditions compares with the repeatability standard deviation
specified in the test method; see test result. E2282
estimated from all laboratories for a particular material (see
3.2 Definitions of Terms Specific to This Standard:
A1.2.3).
3.2.1 average of the cell averages, x%, n—the average of the
cell averages for a particular material.
4. Significance and Use
3.2.2 between-laboratory consistency statistic, h, n—the
4.1 ASTM regulations require precision statements in all
ratio of the cell deviation to the standard deviation of the cell
test methods in terms of repeatability and reproducibility. This
averages.
practice may be used in obtaining the needed information as
3.2.2.1 Discussion—This statistic is an indicator of how one simply as possible. This information may then be used to
E691 − 23
prepare a precision statement in accordance with Practice
Laboratories 9
Materials 10
E177. Knowledge of the test method precision is useful in
Number of Test Results per Material 11
commerce and in technical work when comparing test results
Protocol 12
against standard values (such as specification limits) or be-
Conducting the Testing Phase of the ILS Section
tween data sources (different laboratories, instruments, etc.).
Pilot Run 13
4.1.1 When a test method is applied to a large number of
Full Scale Run 14
portions of a material that are as nearly alike as possible, the
Calculation and Display of Statistics Section
test results obtained will not all have the same value. A
Calculation of the Statistics 15
measure of the degree of agreement among these test results
Tabular and Graphical Display of Statistics 16
describes the precision of the test method for that material.
Data Consistency Section
Numerical measures of the variability between such test results
Flagging Inconsistent Results 17
provide inverse measures of the precision of the test method.
Investigation 18
Greater variability implies smaller (that is, poorer) precision Task Group Actions 19
Glucose ILS Consistency 20
and larger imprecision.
4.1.2 Precision is reported as a standard deviation, coeffi-
Precision Statement Information Section
cient of variation (relative standard deviation), variance, or a Repeatability and Reproducibility 21
precision limit (a data range indicating no statistically signifi-
Section
cant difference between test results).
Keywords 22
4.1.3 This practice is designed only to estimate the precision
Tables Table
of a test method. However, when accepted reference values are
Glucose in Serum Example 1–4, 6–8
available for the property levels, the test result data obtained
Critical Values of Consistency Statistics, h and k 5
according to this practice may be used in estimating the bias of
Figures Figure
the test method. For a discussion of bias estimation and the
Glucose in Serum Example 1–3
relationships between precision, bias, and accuracy, see Prac-
Annexes Annex
tice E177.
Theoretical Considerations Annex A1
4.2 The procedures presented in this practice consist of Calculation of the ILS Statistics for Unbalanced Data Sets Annex A2
three basic steps: planning the interlaboratory study, guiding
Appendixes Appendix
the testing phase of the study, and analyzing the test result data.
Spreadsheet for E691 Calculations Appendix X1
4.2.1 The planning phase includes forming the ILS task
group, the study design, selection, and number of participating
5. Concepts of Test Method Precision
laboratories, selection of test materials, material certifications
5.1 Repeatability and Reproducibility—These two terms
if applicable, and writing the ILS protocol. A well-developed
deal with the variability of test results obtained under specified
test method is essential, so including a ruggedness test to
laboratory conditions and represent the two extremes of test
determine control of test method conditions is highly recom-
method precision. Repeatability concerns the variability be-
mended.
tween independent test results obtained within a single labo-
NOTE 1—In this practice, the term test method is used both for the actual
ratory in the shortest practical period of time by a single
measurement process and for the written description of the process, while
operator with a specific set of test apparatus using test
the term protocol is used for the directions given to the laboratories for
specimens (or test units) taken at random from a single quantity
conducting the ILS.
of homogeneous material obtained or prepared for the ILS.
4.2.2 The testing phase includes material preparation and
Reproducibility deals with the variability between single test
distribution, liaison with the participating laboratories, and
results obtained in different laboratories, each of which has
handling of test result data received from the laboratories.
applied the test method to test specimens (or test units) taken
4.2.3 The data analysis utilizes tabular, graphical, and sta-
at random from a single quantity of homogeneous material
tistical diagnostic tools for evaluating the consistency of the
obtained or prepared for the ILS.
data so that unusual values may be detected and investigated,
and also includes the calculation of the numerical measures of 5.1.1 Repeatability Conditions—The single-operator,
precision of the test method pertaining to repeatability and single-set-of-apparatus requirement means that for a particular
reproducibility. step in the measurement process the same combination of
operator and apparatus is used for every test result and on every
4.3 The information in this practice is arranged as follows:
material. Thus, one operator may prepare the test specimens, a
Section
second measure the dimensions and a third measure the
Scope 1
Referenced Documents 2 breaking force. “Shortest practical period of time” means that
Terminology 3
the test results, at least for one material, are obtained in a time
Significance and Use 4
not less than in normal testing and not so long as to permit
Concepts of Test Method Precision 5
significant changes in test material, equipment or environment.
Planning the Interlaboratory Study (ILS) Section
5.1.2 Reproducibility Conditions—The factors that contrib-
ILS Membership 6
Basic Design 7 ute to variability in a single laboratory, such as operator,
Test Method 8
equipment used, calibration of the equipment, and environment
E691 − 23
(for example, temperature, humidity, air pollution) will gener- results for each material. In addition, it should specify any
ally have different effects in other laboratories, and the vari- special calibration procedures and the repeatability conditions
ability among laboratories will be greater. to be specified in the protocol (see 12.3 and 12.4).
5.2 Observations, Test Determinations, and Test Results—A
6.2 ILS Coordinator—The task group must appoint one
test method often has three distinct stages: the direct observa-
individual to act as overall coordinator for conducting the ILS.
tion of dimensions or properties, the arithmetic combination of The coordinator will supervise the distribution of materials and
the observed values to obtain a test determination, and the
protocols to the laboratories and receive the test result reports
arithmetic combination of a number of test determinations to from the laboratories. Scanning the reports for gross errors and
obtain the test result of the test method.
checking with the laboratories, when such errors are found,
5.2.1 In the simplest of test methods a single direct obser- will also be the responsibility of the coordinator. The coordi-
vation is both the test determination and the test result. For
nator may wish to consult with the statistician in questionable
example, the test method may require the measurement of the cases.
length of a test specimen dimension, which then becomes the
6.3 Statistician:
test result.
6.3.1 The test method task group should obtain the assis-
5.2.2 A test determination may involve a combination of
tance of a person familiar with the statistical procedures in this
two or more observations. For example, a test method may
practice and with the materials being tested in order to ensure
require the measurement of the mass and the volume of the test
that the requirements outlined in this practice are met in an
specimen, and then direct that the mass be divided by the
efficient and effective manner. This person should also assist
volume to obtain the density of the specimen. The whole
the task group in interpreting the results of the data analysis.
process of measuring the mass and the volume, and calculating
6.3.2 When a person having adequate knowledge of both
the density, is a test determination.
the materials and the proper statistical techniques is not
5.2.2.1 If the test method specifies that only one test
available, the task group should obtain the services of a
determination is to be made, then the test determination value
statistician who has experience in practical work with data
is the test result of the test method. Some test methods require
from materials testing. The task group should provide the
that several determinations be made and the values obtained be
statistician with an opportunity to become familiar with the
averaged or otherwise combined to obtain the test result of the
statistical procedures of this practice and with both the mate-
test method. Averaging of several determinations is often used
rials and the test method involved. The statistician should
to reduce the effect of local variations of the property within
become a member of the task group conducting the ILS (task
the material.
group members need not be members of ASTM).
5.2.2.2 In this practice, the term test determination is used
6.3.3 The calculations of the statistics (see Section 15) for
both for the process and for the value obtained by the process,
each material can be readily done by persons not having
except when test determination value is needed for clarity.
statistical knowledge (see 15.1.3 and 15.4.2).
5.2.3 The test result is the final reportable value of the test
method. The precision of a test method is determined from test
6.4 Data Analyst—This individual should be someone who
results, not from test determinations or observations.
is careful in making calculations and can follow the directions
5.2.3.1 The number of test results conducted by each
in Sections 15 through 17.
laboratory on a material that is required for an interlaboratory
6.5 Laboratory ILS Supervisor—Each laboratory must have
study of a test method is specified in the protocol of that study.
an ILS supervisor to oversee the conduct of the ILS within the
5.2.4 Test Specimens and Test Units—In this practice a test
laboratory and to communicate with the ILS Coordinator. The
unit is the total quantity of material needed for obtaining a test
name of the supervisor should be obtained on the response
result as specified by the test method. The portion of the test
form to the “invitation to participate” (see 9.4).
unit needed for obtaining a single test determination is called a
test specimen. Usually a separate test specimen is required for
7. Basic Design
each test determination.
7.1 Keep the design as simple as possible in order to obtain
PLANNING THE INTERLABORATORY STUDY
estimates of within- and between-laboratory variability that are
(ILS)
free of secondary effects. The basic design is represented by a
two-way classification table in which the rows represent the
6. ILS Membership
laboratories, the columns represent the materials, and each cell
6.1 Task Group —Either the task group that developed the
(that is, the intersection of a row with a column) contains the
test method, or a special task group appointed for the purpose,
test results made by a particular laboratory on a particular
must have overall responsibility for the ILS, including funding
material (see Table 1).
where appropriate, staffing, the design of the ILS, and decision-
making with regard to questionable data. The task group
8. Test Method
should decide on the number of laboratories, materials, and test
8.1 Of prime importance is the existence of a valid, well-
written test method that has been developed in one or more
competent laboratories. It is highly recommended that the test
To facilitate the preparation of the final report on the ILS, the task group can
obtain the Research Report format guide from ASTM Headquarters. method be subjected to a ruggedness test prior to the ILS.
E691 − 23
8.2 A ruggedness test is a screening procedure for investi- time and interest to do a good job. If a laboratory meets all the
gating the effects of variations in environmental or other other requirements, but has had insufficient experience with the
conditions in order to determine how control of such test test method, the operator in that laboratory should be given an
conditions should be specified in the written description of the opportunity to familiarize himself with the test method and
method. For example, the temperature of the laboratory or of a practice its application before the ILS starts. For example, this
heating device used in the test may have an effect that cannot experience can be obtained by a pilot run (see Section 13)
be ignored in some cases but may be much less in others. In a using one or two trial samples provided by the task group and
ruggedness test, deliberate variations in temperature would be returning the raw data and the test results to the task group.
introduced to establish the allowable limits on control of The importance of this familiarization step cannot be
temperature. This subject is discussed more fully in Practice overemphasized. Many interlaboratory studies have turned
E1169. out to be essentially worthless due to lack of familiarization.
8.3 As a result of carrying out the screening procedure, and 9.4 Obtain written ensurance from each potential participat-
of some experience with the test method in the sponsoring
ing laboratory that it is properly equipped to follow all the
laboratory and one or two other laboratories, a written version details of the procedure and is willing to assign the work to a
of the test method must have been developed (but not neces-
skilled operator in a timely manner. The decision of a labora-
sarily published as a standard method). This draft should
tory to participate should be recorded on a response form to a
describe the test procedure in terms that can be easily followed
written invitation. The invitation should include information
in any properly equipped laboratory by competent personnel
covering the required time for calibrating the apparatus and for
with knowledge of the materials and the property to be tested.
testing all of the materials, and other possible costs. The
The test conditions that affect the test results appreciably
response form should include the name, address, and telephone
should have been identified and the proper degree of control of
number of the person supervising the ILS work within the
the test conditions specified in the description of the test
laboratory, the address and other markings required to ensure
procedure. In addition, the test method should specify how
the ILS sample material will be promptly delivered to the ILS
closely (that is, to how many digits) each observation in the test
supervisor, answers to brief questions concerning equipment,
method is to be measured.
environment, and personnel, including previous use of the test
method, upon which the apparent competence of the laboratory
8.4 The test method should specify the calibration proce-
may be judged, and an affirmation that the laboratory under-
dure and the frequency of calibration.
stands what is involved and agrees to carry out its responsi-
bilities with diligence.
9. Laboratories
9.5 The ILS should not be restricted to a group of labora-
9.1 Number of Laboratories:
tories judged to be exceptionally qualified and equipped for the
9.1.1 An ILS should include 30 or more laboratories but this
ILS. Precision estimates for inclusion in a test method should
may not be practical and some ILS have been run with fewer.
be obtained through the efforts of qualified laboratories and
It is important, that enough laboratories be included in the ILS
personnel operating under conditions that will prevail when the
to be a reasonable cross-section of the population of qualified
test method is used in practice.
laboratories; that the loss or poor performance of a few will not
be fatal to the study, and to provide a reasonably satisfactory
10. Materials
estimate of the reproducibility.
9.1.2 Under no circumstances should the final statement
10.1 Material designates anything with a property that can
of precision of a test method be based on acceptable test
be measured. Different materials having the same property may
results for each material from fewer than 6 laboratories.
be expected to have different property levels, meaning higher
This would require that the ILS begin with 8 or more
or lower values of the property. Different dilutions of the same
laboratories in order to allow for attrition.
material or compound to be assayed are considered “different
9.1.3 The examples given in this practice include only 8 and
materials” for the purpose of this practice. The terminology
7 laboratories, respectively. These numbers are smaller than
“different levels of material” may be used, if appropriate.
ordinarily considered acceptable, but they are convenient for
10.2 The number and type of materials to be included in an
illustrating the calculations and treatment of the data.
ILS will depend on the range of the levels in the class of
9.2 Any laboratory considered qualified to run the test
materials to be tested and likely relation of precision to level
routinely (including laboratories that may not be members of
over that range, the number of different types of materials to
ASTM) should be encouraged to participate in the ILS, if the
which the test method is to be applied, the difficulty and
preparatory work is not excessive and enough suitably homo-
expense involved in obtaining, processing, and distributing
geneous material is available. In order to obtain an adequate
samples, the difficulty of, length of time required for, and
number of participating laboratories, advertise the proposed
expense of performing the test, the commercial or legal need
ILS in where appropriate (for example, trade magazines,
for obtaining a reliable and comprehensive estimate of
meetings, circulars, etc.).
precision, and the uncertainty of prior information on any of
these points.
9.3 “Qualified” implies proper laboratory facilities and test-
ing equipment, competent operators, familiarity with the test 10.2.1 For example, if it is already known that the precision
method, a reputation for reliable testing work, and sufficient is either relatively constant or proportional to the average level
E691 − 23
over the range of values of interest, a smaller number of apparatus or procedure, the protocol should specify which
materials will be needed than if it is merely known that the option or options have been selected for the ILS. Test units and
precision is different at different levels. The ruggedness test
test data sheets must be provided for each option.
(see 8.2) and the preliminary pilot program (see Section 13)
12.3 When special calibration procedures are required be-
help to settle some of these questions, and may often result in
fore every determination or every test result, they should be
the saving of considerable time and expense in the full ILS.
described specifically in the test method. If the test method
10.2.2 An ILS of a test method should include at least three
specifies calibration only daily or less frequently, the ILS task
materials representing different test levels, and for develop-
group must decide whether to require recalibration before
ment of broadly applicable precision statements, six or more
obtaining each test result. While doing so will eliminate
materials should be included in the study.
calibration drift and help ensure relative independence of the
10.2.3 The materials involved in any one ILS should differ
test results, changes in calibration may increase the variability
primarily only in the level of the property measured by the test
between test results.
method. When it is known, or suspected, that different classes
of materials will exhibit different levels of precision when
12.4 Describe any special circumstances that must be ad-
tested by the test method, consideration should be given to
dressed in implementing the repeatability conditions, such as
conducting separate interlaboratory studies for each class of
the period of time between obtaining the test results for the
material.
same material; that is, not less than in normal testing and not so
long as to likely permit significant changes in test material,
10.3 Each material in an ILS should be made to be or
equipment or environment.
selected to be as homogeneous as possible prior to its subdi-
vision into test units or test specimens. If the randomization
12.5 Specify the requirements for acquisition, shipment,
and distribution of individual test specimens (rather than test
documentation (including any material certifications), care,
units) does not conflict with the procedure for preparing the
handling, and conditioning of the materials to be tested.
sample for test, as specified in the test method, greater
Explain the coding system used in identifying the materials and
homogeneity between test units can be achieved by randomiz-
the distinction between test units and test specimens, where
ing test specimens. Then each test unit would be composed of
appropriate.
the required number of randomized test specimens. (See
Section 11 and 14.1 for the quantity of each material needed, 12.6 Supply data sheets for each material for recording the
its preparation and distribution.) raw data as observations are made. Give instructions on the
number of significant digits to be recorded, usually one more,
NOTE 2—It may be convenient to use established reference materials,
if possible, than required by the test method. Also, supply test
since their homogeneity has been demonstrated.
result sheets on which test results can be calculated and
11. Number of Test Results per Material
reported. In many instances this can be combined with the raw
data sheet. Specify the number of significant digits to be
11.1 In the design of an ILS a sufficient total number of test
reported, usually two more than required by the test method.
results on each material must be specified to obtain a good
Request the laboratories send raw data and test result sheets as
estimate of the measure of repeatability, generally the repeat-
soon as the testing is completed, and at least weekly if testing
ability standard deviation. In many cases, the standard devia-
tion in question will be a function of the property level being will continue over several weeks. For guidance on the number
of significant digits needed for data reporting see Practice E29.
measured. When this occurs, the standard deviation should be
determined separately for each level. It is generally sound to
12.7 Request that each laboratory keep a record (or log) of
limit the number of test results on each material in each
any special events that arise during any phase of the testing.
laboratory to a small number, such as three or four. The
This record, to be sent to the ILS coordinator, will provide a
minimum number of test results per laboratory will normally
valuable source of information both in dealing with unusual
be three for a chemical test and three or four for a physical or
data and in making improvements in the test method in future
optical test. The number may be as small as two when there is
revisions.
little danger that a test unit will be lost or questionable test
12.7.1 Instruct the laboratories to notify the ILS coordinator
results obtained, or as many as ten when test results are apt to
promptly whenever an error in test procedure arises, so that a
vary considerably. Generally, the time and effort invested in an
decision can be made as to whether a new set of test units
ILS is better spent on examining more materials across more
should be sent to the laboratory for a complete retest of the
laboratories than on recording a large number of test results per
material.
material within a few laboratories.
12.8 Enclose with the protocol a questionnaire requesting
12. Protocol
information on specific aspects of the apparatus, reagents,
12.1 In the protocol, cite the name, address, and telephone
calibration, or procedure, as well as any other information that
number of the person who has been designated ILS coordinator
might assist in dealing with data inconsistencies, or ensure the
(see 6.2). Urge the laboratories to call the coordinator when
task group that the laboratory complied with the current
any questions arise as to the conduct of the ILS.
requirements of the test method. Also obtain any other infor-
mation that may be needed in preparing the final research
12.2 Clearly identify the specific version of the test method
being studied. If the test method allows several options in report on the ILS.
E691 − 23
CONDUCTING THE TESTING PHASE 14.1.5 Replacement Sets of Test Units—As the ILS
OF THE ILS progresses, a laboratory may discover that the test method was
not used properly on some test units. The laboratory ILS
13. Pilot Run supervisor should discuss this with the ILS coordinator, who
may send a replacement set of test units, replace the misused
13.1 Before investing laboratory time in the full scale ILS,
test units, or do nothing, as may seem desirable.
it is usually wise to conduct a pilot run with only one, or
14.2 Checking Progress—From time to time, at intervals
perhaps two, material(s) to determine whether the test method
appropriate to the magnitude of the ILS, the coordinator should
as well as the protocol and all the ILS procedures are clear, and
call each ILS supervisor to ascertain how the testing is
to serve as a familiarization procedure for those without
progressing. By comparing the progress of all laboratories, the
sufficient experience with the method (see 9.3). The results of
coordinator can determine whether some laboratories are
this pilot run also give the task group an indication of how well
lagging considerably behind the others and so advise these
each laboratory will perform in terms of promptness and
laboratories.
following the protocol. Laboratories with poor performance
should be encouraged and helped to take corrective action.
14.3 Data Inspection—The completed data sheets should be
examined by the coordinator immediately upon receipt in order
13.2 All steps of the procedures described in this practice
to detect unusual values or other deficiencies that should be
should be followed in detail to ensure that these directions are
questioned. Replacement sets of test units or of specific test
understood, and to disclose any weaknesses in the protocol or
units may be sent when there is missing or obviously erroneous
the test method.
data. The task group can decide later whether or not the
additional data should be used in the estimation of the precision
14. Full Scale Run
of the test method.
14.1 Material Preparation and Distribution:
CALCULATION AND DISPLAY OF STATISTICS
14.1.1 Sample Preparation and Labelling—Prepare enough
of each material to supply at least 10 % more than needed by
15. Calculation of the Statistics
the number of laboratories committed to the ILS. Label each
test unit or test specimen with a letter for the material and a
15.1 Overview—The analysis and treatment of the ILS test
sequential number. Thus, for ten laboratories and two test
results have three purposes, to determine whether the collected
results for each laboratory the test units for Material B would
data are adequately consistent to form the basis for a test
be numbered from B1 to B22, or, if five test specimens per test
method precision statement, to investigate and act on any data
unit are required, the test specimens may be numbered B1 to
considered to be inconsistent, and to obtain the precision
B110.
statistics on which the precision statement can be based. The
14.1.2 Randomization—For each material independently,
statistical analysis of the data for estimates of the precision
allocate the specified number of test units or test specimens to
statistics is simply a one-way analysis of variance (within- and
each laboratory, using a random number table, or a suitable
between-laboratories) carried out separately for each level
computerized randomization based on random numbers. See
(material). Since such an analysis can be invalidated by the
Guide E1402 for a discussion of randomization.
presence of severe outliers, it is necessary to first examine the
14.1.3 Shipping—Ensure that the test units are appropriately
consistency of the data. Annex A1 gives background theory on
packaged to arrive in the desired condition. When the material
these procedures. The following paragraphs show, in terms of
is sensitive to the conditions to which it is exposed (vibrations,
a numerical example, how the entire program is carried out:
light, heat, humidity, etc.), place special directions for opening
15.1.1 The calculations are illustrated with test results from
the package on a label outside the package. If needed, have the
an ILS in which the concentration of glucose in serum (see
shipper monitor any specified factors deemed to be important
Table 1) was measured at five different concentration levels by
from the point of origin to the final destination of any
eight laboratories. Each laboratory obtained three test results at
packages. The monitoring of shipments can be done as well by
each concentration level.
including ILS’s own electronic shipment monitoring device.
15.1.2 For extended calculations it is usually necessary to
Clearly indicate the name of the person who has been desig-
retain extra significant digits in order to ensure that statistically
nated as ILS supervisor at the laboratory on the address of each
important information is not lost in calculation by rounding off
package. Follow each laboratory’s instructions for ensuring
too soon. As a general rule, retain at least two more digits in the
prompt delivery of the package. Include any Material Certifi-
averages than in the reported test results and at least three
cations required with the materials shipped to the laboratories.
significant figures in the standard deviations.
14.1.4 Follow-Up—Once the test units have been shipped, 15.1.3 While the calculations described in this section are
the ILS coordinator should call each laboratory ILS supervisor arranged for use of a hand calculator, they also can be readily
within a week to ten days to confirm that all test units have programmed for the computer. A spreadsheet can be easily
arrived safely. It is important for the ILS coordinator to express adapted to these calculations, and Appendix X1 illustrates an
the need for the laboratory ILS supervisor to ensure that only example spreadsheet for the glucose in serum ILS.
the correct number of replicates are tested and that the test 15.1.4 If laboratory data contains either missing or an
results are reported to the number of decimal places as required excessive number of test results than required by the protocol,
in the protocol. this will result in an unbalanced data set for that material. In
E691 − 23
TABLE 1 Glucose in Serum ILS Test Result Data
given in Annex A2. The consistency statistics must be adjusted
Material for the data imbalance. A highly unbalanced data set, with a
Laboratory
A B C D E deviation of 10 % or greater from the targeted number of
1 41.03 78.28 132.66 193.71 292.78 required test results, can lead to much greater variability in the
41.45 78.18 133.83 193.59 294.09
estimates of precision.
41.37 78.49 133.10 193.65 292.89
15.2 Table of ILS Test Results—The test results received
2 41.17 77.78 132.92 190.88 292.27
from the laboratories are usually best arranged in rows and
42.00 80.38 136.90 200.14 309.40
columns as in Table 1. Each column contains the data obtained
41.15 79.54 136.40 194.30 295.08
from all laboratories for one material, and each row contains
3 41.01 79.18 132.61 192.71 295.53
the data from one laboratory for all materials. The test results
40.68 79.72 135.80 193.28 290.14
from one laboratory on one material constitute a cell. Thus, the
42.66 80.81 135.36 190.28 292.34
cell for Laboratory 2 and Material C contains the test results
4 39.37 84.08 138.50 195.85 295.19
132.92, 136.90, and 136.40. This cell is called C2, by material
42.37 78.60 148.30 196.36 295.44
and laboratory. It helps in the interpretation of the data to
42.63 81.92 135.69 199.43 296.83
arrange the materials in increasing order of the measured
5 41.88 78.16 131.90 192.59 293.93
values.
41.19 79.58 134.14 191.44 292.48
41.32 78.33 133.76 195.12 294.28
15.3 Worksheets—Generally, it facilitates the calculations to
prepare a separate calculation worksheet for each material,
6 43.28 78.66 137.21 195.34 297.74
using Table 2 as a model but making appropriate changes for
40.50 79.27 135.14 198.26 296.80
42.28 81.75 137.50 198.13 290.33
different numbers of laboratories, and test results per material.
Enter the test result data for one material (from one column of
7 41.08 79.76 130.97 194.66 287.29
Table 1) on a worksheet. Also enter the results of the following
41.27 81.45 131.59 191.99 293.76
39.02 77.35 134.92 187.13 289.36
calculations for that material on the same worksheet, as
illustrated in Table 2. Work on only one material at a time.
8 43.36 80.44 135.46 197.56 298.46
42.65 80.80 135.14 195.99 295.28
15.4 Cell Statistics:
41.72 79.80 133.63 200.82 296.12
15.4.1 Cell Average, x¯—Calculate the cell average for each
laboratory using the following equation:
n
this situation, the calculations in this section cannot be used,
x¯ 5 x/n (1)
(
but a methodology for calculating the precision statistics is 1
TABLE 2 Interlaboratory Study Worksheet for Glucose in Serum Initial Preparation of Test Result Data for Material C
Test Results, x
Laboratory
s d h k
x¯
Number
1 2 3
1 132.66 133.83 133.10 133.197 0.591 –1.946 –0.73 0.22
2 132.92 136.90 136.40 135.407 2.168 0.264 0.10 0.79
3 132.61 135.80 135.36 134.590 1.729 –0.553 –0.21 0.63
4 138.50 148.30 135.69 140.830 6.620 5.687 2.14 2.41
5 131.90 134.14 133.76 133.267 1.199 –1.876 –0.71 0.44
6 137.21 135.14 137.50 136.617 1.287
...