ASTM C1067-20

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: C1067 − 12 C1067 − 20
Standard Practice for
Conducting a Ruggedness Evaluation or Screening Program
for Test Methods for Construction Materials
This standard is issued under the fixed designation C1067; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope*
1.1 This practice covers a procedure for evaluating the ruggedness of a test method by determining the effects of different
experimental factors on the variation of test results. The procedure is intended for use during the development of a test method
before the interlaboratory study is executed, such as those described in Practices C802 and E691.
1.2 This practice covers, in general terms, techniques for planning, collecting data, and analyzing results from a few laboratories.
Appendix X1 provides the details of the procedure with an example and Appendix X2 provides additional information on the
methodology.
1.3 The practice is not intended to give information pertinent to estimating multilaboratory precision.
1.4 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only in illustrations
of calculation methods.
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety safety, health, and healthenvironmental practices and determine the
applicability of regulatory limitations prior to use.
1.6 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
C670 Practice for Preparing Precision and Bias Statements for Test Methods for Construction Materials
C802 Practice for Conducting an Interlaboratory Test Program to Determine the Precision of Test Methods for Construction
Materials
E456 Terminology Relating to Quality and Statistics
E691 Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method
E1169 Practice for Conducting Ruggedness Tests
This practice is under the jurisdiction of ASTM Committee C09 on Concrete and Concrete Aggregates and is the direct responsibility of Subcommittee C09.94 on
Evaluation of Data (Joint C09 and C01).
Current edition approved July 1, 2012April 15, 2020. Published September 2012August 2020. Originally approved in 1987. Last previous edition approved in 20072012
as C1067 – 00 (2007). 12. DOI: 10.1520/C1067-12.10.1520/C1067-20.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
*A Summary of Changes section appears at the end of this standard
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
C1067 − 20
3. Terminology
3.1 Definitions:
3.1.1 For definitions of statistical terms used in this standard, refer to Terminology E456.
3.2 Definitions of Terms Specific to This Standard:
3.2.1 determination, n—numerical value of a characteristic of a test specimen measured in accordance with the given the test
method.
3.2.2 effect, n—of a factor, the difference in the measured characteristics at each level of a factor averaged over all levels of other
factors in the experiment.
3.2.3 factor, n—a condition or element in the test procedure or laboratory environment that can be controlled and that is a potential
source of variation of determinations.test results.
3.2.4 level, n—the value or setting of a factor associated with a determination.
3.2.5 replication, n—the act of obtaining, under specified conditions, two or more determinations on identical specimens.
3.2.5.1 Discussion—
Replicate determinations are typically required to be obtained by the same operator, using the same apparatus, on specimens that
are as similar as possible, and during a short time interval.
3.2.6 ruggedness, n—the characteristic of degree to which a test method such that determinations are not influencedis able to
produce test results that are not influenced, to a statistically significant degree, by small changes in the testing procedure or
environment.
3.2.6.1 Discussion—
Statistical significance is evaluated by comparing the observed variation in test results due to a factor towith the expected variation
due to chance alone.
3.2.7 screening, n—a planned experiment using a low number of determinations to detect among many factors those that have a
statistically significant effect on variation of determinations test results compared with chance variation.
3.2.7.1 Discussion—
In this practice, the influence of seven factors is evaluated using a replicated set of eight determinations, that is, a total of 16
determinations.
4. Summary of Practice
4.1 The practice requires that the user develop, from theoretical or practical knowledge, or both, a list of factors that plausibly
would cause statistically significant variation in test results (determinations) if the factors were not controlled. The technique is
limited to the analysis of the effects of seven factors and requires ⁄16 of the determinations that would be required to evaluate seven
factors in a full factorial study. study (see Appendix X2 for additional explanation). Procedures exist for analysis of smaller and
larger numbers of factors (see Guide E1169), but seven is a convenient number for many test methods for construction materials.
The seven-factor analysis requires 16 determinations by each laboratory. The procedure can be executed usefully by a single
laboratory, but sometimes additional information can be obtained if it is repeated in one or two additional laboratories.
4.2 The procedure requires that two levels of each factor be identified, and 16 determinations be obtained with prescribed
combinations of factor levels. The levels assigned to a factor may be quantitative or qualitative (for example, 20°C versus 25°C
or brass versus steel).
4.3 After data are acquired, a statistical procedure is applied to establish which of the factors under study have a statistically
significant effect on test results.
5. Significance and Use
5.1 The purpose of a ruggedness evaluation, or screening program, is to determine the sensitivity of the test method to changes
C1067 − 20
in levels of pertinent operating factors using a small number of tests. Normally, operating conditions for a test method are defined
along with allowable tolerances. A ruggedness analysis determines the effect of “worst-case” variation in operating conditions
within the specified tolerances. If the ruggedness evaluation indicates high variation (poor precision), the that the factors have a
statistically significant effect on test results, the method can be revised with smaller tolerances on operating conditions to improve
the precision. reduce variation among test results.
5.2 This practice evaluates the effects of seven factors using eight treatments. testing conditions (treatments). The disadvantage
of this approach is that it only estimates the main effects of the factors and does not detect the effects of interactions among factors.
For this reason, this is a screening program and additional investigation is required to investigatedetermine whether there are
interaction effects.
5.3 A major reason for poor precision in test methods is the lack of adequate control over the sources of variation in testing
procedures or testing environments. These sources of variation often are not controlled adequately because they were not identified
during the development of the test procedures as having a large effect on the determinations.test results. This practice provides a
systematic procedure to establish the required degree of control for different testing parameters.
5.4 All new test methods must be subjected to an interlaboratory program to develop a precision and bias statement. These
programs can be expensive and lengthy,time-consuming, and the result may show that the method is too variable and should not
be published without further revision. Interlaboratory studies may give the subcommittee an indication that the method is too
variable, but they do not usually give a clear picture of the causes of the high variation. Application of this practice using one or
two laboratories before finalizing the test method and conducting the interlaboratory study is an economical way to determine these
causes.
5.5 Many existing test methods were developed before there was a requirement for precision and bias statements. Since this
became a requirement, most of these test methods have developed precision and bias statements, and the result is that many have
been found to suffer from relatively large amount of variation. This practice provides a relatively simple and economical way to
investigate the causes of variation in test methods, so that a subcommittee will have some guidance as to which parts of the test
method need to be revised.
5.6 The procedure can be used for a screening program within a single laboratory, but involvement of at least three laboratories
is recommended, particularly if the single laboratory were to be the one that developed the test method. This is particularly
important for new test methods. The originating laboratory is so much a part of the development of the test method that it is difficult
for it to be objective in spotting any problems in the clarity of the test method directions.instructions. Two additional laboratories
will probably contribute fresh critical review of the validity of the test method and provide assistance in clarifying the instructions
of the test method whenif needed. This practice, however, is not intended to provide information on multilaboratory precision, but
it does provide some information on single-operator precision, which could be used to develop a temporary repeatability statement
until the interlaboratory study is completed.
6. Materials
6.1 The number and types of material shall cover the range of material properties to which the test method is applicable. The test
method may not apply to material types or property values outside the range evaluated. Three to five materials with different
properties values of the measured property will usually be sufficient.
6.1.1 Some preliminary testing may help the laboratories involved determine the materials that will be used in the screening
program.
7. Procedure
7.1 Determine the number of laboratories that will participate in the screening program and which materials each will use. The
maximum amount of information is obtained if all laboratories include all materials in their part of the program, however, cost can
be reduced if each laboratory uses a different material. In this case, caution must be exercised in interpreting the results because
laboratory-dependent effects cannot be separated from material-dependent effects.
7.2 Factors that are likely to have the greatest effect on the variability of the determinations test results are selected for study.
Levels of these factors are determined by selecting the minimum and maximum levels that would plausibly occur in the execution
C1067 − 20
A
TABLE 1 Pattern of Assigning Levels to Seven Factors
Determination
Number
Factor
A B C D E F G
B
1 (9) a b c D E F g
2 (10) a b C D e f G
3 (11) a B c d E f G
4 (12) a B C d e F g
5 (13) A b c d e F G
6 (14) A b C d E f g
7 (15) A B c D e f g
8 (16) A B C D E F G
A
TABLE 1 Pattern of Assigning Levels to Seven Factors
Determination
Number
Factor
B
1(9) 2(10) 3(11) 4(12) 5(13) 6(14) 7(15) 8(16)
A – – – – + + + +
B – – + + – – + +
C – + – + – + – +
D + + – – – – + +
E + – + – – + – +
F + – – + + – – +
G – + + – + – – +
A
Lower case letter The plus sign (+) indicates one level for the factor and upper
case letter the minus sign (–) indicates the other level.
B
The numbers in parentheses refer to the determinations in replicate set 2.
of the test method if there were no particular efforts to control them. Levels often represent quantitative factors, such as temperature
or pressure, but they may also represent qualitative factors, such as old versus new or wet versus dry. Only two levels are allowed
for each factor. In this practice, factors are assigned letter designations, A through G, and the two levels of each factor are
designated with upper and lower cases of these letters, plus (+) and minus (–) signs, as shown in Table 1.
NOTE 1—In textbooks dealing with design of experiments, factor levels are often denoted with plus (+) and minus (-) signs.
7.3 Assign combinations of factor levels to each determination according to Table 1. The eight determinations will be replicated;
therefore, the full study on each material will require 16 determinations. Run the 16 determinations in random order.
7.4 To analyze the results, construct a 16 row by 16 column results matrix composed of 61 values as shown in Table 2. Each of
the 16 columns corresponds to one of the determinations. The values in row 1 are all +1. The +1 and –1 values in rows 2 to 8 for
each the first replicate set correspond to the high and low settings of the factors A through G as given in Table 1. The pattern in
rows 1 to 8 of the first replicate set is repeated for rows 1 to 8 of the second replicate set and for rows 9 to 16 of the secondfirst
replicate set. For rows 9 to 16 of the second replicate set, the signs are reversed from those in the first set. The various combinations
of plus and minus valuessigns in Table 2 are applied to the values of the 16 determinations to create a table of signed determinations
and various sums of the signed determinations are calculated. calculated (see Note 1). For each row of Table 2, the table of signed
determinations, calculate the Z and W statistics using Eq 1 and 2.
Z 5 α d (1)
r ( ri i
i51
Z
r
W 5 (2)
r
where:
r = row number as shown in Table 2, where r = 1 to 16,
i = determination number ranging from 1 to 16,
α = +1 or -1 as defined in Table 2 for each row number and determination number, and
ri
α = +1 or –1 as defined in Table 2 for each row number and determination number, and
ri
d = measured value of determination number i as defined in Table 1.
i
NOTE 1—See Table X1.4 for an example of the resulting table of signed determinations after the +1 and –1 values shown in Table 2 are applied to the
16 determinations. Eq 1 represents the sum for each row of the table of signed determinations.
C1067 − 20
TABLE 2 Matrix of Signs to be Applied to 16 Determinations (d to d ) to Calculate Z- and W-Statistics
1 16
Sign Applied to Each Determination in Computing Z
i
Eight Determinations for Replicate Set 1 Eight Determinations for Replicate Set 2
row 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Z W
A
Row 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Z W
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Z W
1 1
2 -1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1 1 1 1 1 Z W
2 2
2 –1 –1 –1 –1 1 1 1 1 –1 –1 –1 –1 1 1 1 1 Z W
2 2
3 -1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 1 Z W
3 3
3 –1 –1 1 1 –1 –1 1 1 –1 –1 1 1 –1 –1 1 1 Z W
3 3
4 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 Z W
4 4
4 –1 1 –1 1 –1 1 –1 1 –1 1 –1 1 –1 1 –1 1 Z W
4 4
5 1 1 -1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1 1 1 Z W
5 5
5 1 1 –1 –1 –1 –1 1 1 1 1 –1 –1 –1 –1 1 1 Z W
5 5
6 1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 -1 1 Z W
6 6
6 1 –1 1 –1 –1 1 –1 1 1 –1 1 –1 –1 1 –1 1 Z W
6 6
7 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 Z W
7 7
7 1 –1 –1 1 1 –1 –1 1 1 –1 –1 1 1 –1 –1 1 Z W
7 7
8 -1 1 1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 Z W
8 8
8 –1 1 1 –1 1 –1 –1 1 –1 1 1 –1 1 –1 –1 1 Z W
8 8
9 1 1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 Z W
9 9
9 1 1 1 1 1 1 1 1 –1 –1 –1 –1 –1 –1 –1 –1 Z W
9 9
10 -1 -1 -1 -1 1 1 1 1 1 1 1 1 -1 -1 -1 -1 Z W
10 10
10 –1 –1 –1 –1 1 1 1 1 1 1 1 1 –1 –1 –1 –1 Z W
10 10
11 -1 -1 1 1 -1 -1 1 1 1 1 -1 -1 1 1 -1 -1 Z W
11 11
11 –1 –1 1 1 –1 –1 1 1 1 1 –1 –1 1 1 –1 –1 Z W
11 11
12 -1 1 -1 1 -1 1 -1 1 1 -1 1 -1 1 -1 1 -1 Z W
12 12
12 –1 1 –1 1 –1 1 –1 1 1 –1 1 –1 1 –1 1 –1 Z W
12 12
13 1 1 -1 -1 -1 -1 1 1 -1 -1 1 1 1 1 -1 -1 Z W
13 13
13 1 1 –1 –1 –1 –1 1 1 –1 –1 1 1 1 1 –1 –1 Z W
13 13
14 1 -1 1 -1 -1 1 -1 1 -1 1 -1 1 1 -1 1 -1 Z W
14 14
14 1 –1 1 –1 –1 1 –1 1 –1 1 –1 1 1 –1 1 –1 Z W
14 14
15 1 -1 -1 1 1 -1 -1 1 -1 1 1 -1 -1 1 1 -1 Z W
15 15
15 1 –1 –1 1 1 –1 –1 1 –1 1 1 –1 –1 1 1 –1 Z W
15 15
16 -1 1 1 -1 1 -1 -1 1 1 -1 -1 1 -1 1 1 -1 Z W
16 16
16 –1 1 1 –1 1 –1 –1 1 1 –1 –1 1 –1 1 1 –1 Z W
16 16
A
Rows 2 to 8 and rows 10 to 16 are associated with the levels of the factors A–G.
7.5 The Z-statistic for row 1 (Z ) represents the sum of the 16 determinations and Z /16 is the overall average of the 16
1 1
determinations. The Z-statistics for rows 2 through 8 (Z through Z ) are related to the effects of each of the seven factors (see Note
2 8
2). These values of Z represent the differences between the sum of the determinations at the high level of the factor and the sum
of the determinations at the low level of the factor. The Z-values are divided by eight to obtain the effect of each factor averaged
of over the levels of the other factors. For example, Z /8 is the average effect of factor B as it is varied from the low level to the
high level.
NOTE 2—A positive value for an effect of a factor means that the response increases as the factor level is changed from its low level to its high level.
The opposite is the case for a negative effect. Recall that an effect of a factor is the difference between the average of the determinations at the high setting
minus the average at the low setting of the factor.
7.6 The W values are various mean squares. W is the mean of the square of the sum of all determinations and is not used in this
analysis. The values W to W are the mean squares for each factor and are compared with the random error (see Note 3). The W
2 8
values for rows 9 through 16 (W to W ) are used to calculate the error variance (s ) according to Eq 3 (see Note 4).
9 16
W
( r
r59
s 5 (3)
NOTE 3—Appendix X2 provides additional information of the meaning of the term “mean squares.”
NOTE 4—The error variance s is the pooled variance of the two replicate determinations for each of the eight conditions.conditions (treatments).
7.7 To establish whether a factor has a statistically significant effect on the results, compute the F statistic for each factor using
Eq 4.
W
r
F 5 (4)
f 2
s
C1067 − 20
TABLE 3 Summary of Statistics for Seven Factors and
Random Error
Factor W F
A W F = W /s
2 A 2
B W F = W /s
3 B 3
C W F = W /s
4 C 4
D W F = W /s
5 D 5
E W F = W /s
6 E 6
F W F = W /s
7 F 7
G W F = W /s
8 G 8
W
W
W
W
12 W
o
r
r59
W 2
13 s 5
W
W
W
where:
F = value of F-statistic for factor f (A through G) for the corresponding row (2 through 8) of Table 2.
f
Table 3 summarizes the calculations given by Eq 3 and 4.
W
r
F 5 (4)
f 2
s
where:
F = value of F-statistic for factor f (A through G) for the corresponding row (2 through 8) of Table 2.
f
Table 3 summarizes the calculations given by Eq 3 and 4.
7.8 An F value that is ≥ 5.32 represents a statistically significant effect for factor f at a probability of not greater than 5 % for
f
drawing an erroneous conclusion.
7.9 An example of an analysis of data representing results on 4 materials from 3 laboratories is presented in Appendix X1.
7.10 If desired, one of the alternative methods discussed in X2.5 of Appendix X2 is permitted for determining which factors have
statistically significant effects.
8. Keywords
8.1 analysis of variance; precision; ruggedness; screening; test method; variation
APPENDIXES
(Nonmandatory Information)
X1. EXAMPLE OF A RUGGEDNESS PROGRAM
X1.1 This appendix describes the procedure for conducting a ruggedness evaluation using as an example a proposed test method
for measuring the viscosity of asphalt. Three laboratories participated in the program.
X1.2 As the first step in the ruggedness evaluation, each of the laboratories critically examined the procedure in the proposed test
method. The objectives of the examination were as follows:
1. To determine if the instructions were clear, concise, and complete,
2. To decide which factors were likely to influence test results and therefore should be included in the study,
C1067 − 20
3. To select materials that covered the range of the property of interest for the range of physical forms of the materials to be
tested, and
4. To determine the proper levels to be evaluated for each of the chosen factors.
X1.3 In this example, representatives of the three laboratories, after familiarizing themselves with the proposed test method, met
and tried to improve the instructions for the viscosity measurement. They selected factors and levels that they believed could affect
the measured viscosities. In a preliminary investigation, one of the laboratories measured viscosity at 24°C, 25°C, and 26°C and
found that there was about a 10 % variation with a change of 1°C. This was considered too large so 24.6 and 25.4°C were selected
as the lower and upper temperature levels for the ruggedness evaluation. In the same manner, the effects of the other factors were
examined and the two levels to be used for each factor were selected. The seven factors selected for the program and their levels
are shown in Table X1.1. The levels of the factors were assigned to each of the eight determinations in accordance with Table 1
from the body of this practice. Table X1.2 shows the testing conditions (or treatments) for each of the eight replicated
determinations.
X1.4 Four materials were selected to cover the range of viscosities to be measured by the test method. For each testing condition,
the viscosities were determined by each of the three laboratories with one replication. Thus each laboratory conducted 16
determinations for each material, for a total of 64 determinations. For each material, the 16 determinations were acquired in random
order. This is a critical part of the program to guard against systematic variations in the testing conditions. The tests results, grouped
by laboratory, are shown in Table X1.3.
X1.5 After the data were obtained, the results for each laboratory-material combination were analyzed independently. Thus in this
program, there are 12 analyses corresponding to each row of data in Table X1.3. To proceed with each analysis, the relevant row
of data from Table X1.3 is copied into 16 rows to create a 16 by 16 matrix. Each column corresponds to a determination and the
value of that determination is repeated 16 times. The numbers in the matrix are multiplied by the corresponding values of +1 or
-1–1 given in Table 2 in the body of this practice. Table X1.4 is an example of the resulting matrix of signed determinations derived
TABLE X1.1 Levels Assigned to Seven Factors
Factor Level
A: Temperature a = 24.6°C
A = 25.4°C
B: Age of viscometer tube b = New
B = Old
C: Applied vacuum c = 310 mmHg
C = 290 mmHg
D: Stirring sample before d = No stirring
charging viscometer D = Stir for 1 minute
E: Angle of viscometer e = 87° from horizontal
E = 90° from horizontal
F: Height of filling f = 6 mm (1 mm above line)
F = 4 mm (1 mm bellow line)
G: Time viscometer held in bath g = 40 min ( 10 min more than specified)
G = 20 min (10 min less than specified)
TABLE X1.1 Levels Assigned to Seven Factors
Factor Level
A: Temperature – = 24.6°C
+ = 25.4°C
B: Age of viscometer tube – = New
+ = Old
C: Applied vacuum – = 310 mmHg
+ = 290 mmHg
D: Stirring sample before – = No stirring
charging viscometer + = Stir for 1 minute
E: Angle of viscometer – = 87° from horizontal
+ = 90° from horizontal
F: Height of filling – = 6 mm (1 mm above line)
+ = 4 mm (1 mm bellow line)
G: Time viscometer held in bath – = 40 min ( 10 min more than specified)
+ = 20 min (10 min less than specified)
C1067 − 20
TABLE X1.2 Conditions for Each Determination
Factor
Determination A
Number Temperature
1 (9) 24.6°C New 310 mmHg 1 min 90º 4 mm 40 min
2 (10) 24.6°C New 290 mmHg 1 min 87º 6 mm 20 min
3 (11) 24.6°C Old 310 mmHg No 90º 6 mm 20 min
4 (12) 24.6°C Old 290 mmHg No 87º 4 mm 40 min
5 (13) 25.4°C New 310 mmHg No 87º 4 mm 20 min
6 (14) 25.4°C New 290 mmHg No 90º 6 mm 40 min
7 (15) 25.4°C Old 310 mmHg 1 min 87º 6 mm 40 min
8 (16) 25.4°C Old 290 mmHg 1 min 90º 4 mm 20 min
TABLE X1.2 Conditions for Each Determination
Determination
Number
Factor 1(9) 2(10) 3(11) 4(12) 5(13) 6(14) 7(15) 8(16)
A-Temperature 24.6 °C 24.6 °C 24.6 °C 24.6 °C 25.4 °C 25.4 °C 25.4 °C 25.4 °C
B-Viscometer New New Old Old New New Old Old
C-Vacuum 310 mmHg 290 mmHg 310 mmHg 290 mmHg 310 mmHg 290 mmHg 310 mmHg 290 mmHg
D-Stirring 1 min 1 min No No No No 1 min 1 min
E-Angle 90° 87° 90° 87° 87° 90° 87° 90°
F-Fill Height 4 mm 6 mm 6 mm 4 mm 4 mm 6 mm 6 mm 4 mm
G-Time in Bath 40 min 20 min 20 min 40 min 20 min 40 min 40 min 20 min
TABLE X1.3 Viscosity Data
Viscosity
Material First Replicate Determination Number Second Replicate Determination Number
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Laboratory 1
1 2370 2258 2355 2185 1825 1845 1820 1830 2320 2275 2350 2380 1840 1850 1825 1820
2 520 495 519 480 401 404 398 402 492 516 490 522 390 408 402 395
3 4205 4006 4191 3846 3212 3284 3185 3221 4200 4160 4130 4020 3218 3180 3280 3280
4 1075 1061 1060 961 803 793 801 805 1050 1070 1015 1000 808 790 795 805
Laboratory 2
1 2350 2240 2335 2165 1805 1825 1800 1810 2280 2310 2400 2120 1825 1806 1809 1812
2 540 515 539 500 421 424 418 422 518 545 524 492 410 425 430 420
3 4235 4036 4121 3876 3242 3314 3117 3250 4250 4142 3960 4205 3310 3112 3240 3117
4 1102 1040 1085 980 820 811 824 828 1110 1125 1040 1050 825 804 816 835
Laboratory 3
1 2390 2278 2375 2205 1845 1865 1840 1850 2400 2268 2350 2250 1860 1850 1870 1845
2 510 485 509 470 391 394 388 392 505 482 510 480 395 390 385 392
3 4200 3975 4160 3816 3190 3246 3150 3200 4180 3990 4140 3890 3200 3180 3220 3195
4 1050 990 1035 930 786 766 775 780 1040 980 1050 970 780 760 785 782
from the data for Material 1 and Laboratory 1 in Table X1.3.
X1.6 After the 16 by 16 matrix with the proper signs applied to each determination has been created, the next step is to calculate
the sum of each row, with due regard to sign, to row to obtain 16 Z-values, which are identified as Z to Z . Table X1.5 shows
1 16
the resulting sums for Laboratory 1 and Material 1. The value Z represents the sum of all viscosities and Z /16 is the overall
1 1
average viscosity for the laboratory-material combination. The value Z represents the difference between the results at the high
level of factor A and at the low level. In this case factor A is temperature, so Z measures the effect of temperature. In the same
manner, Z measures the effect of the factor B, the age of the viscometer. The value Z measures the effect of factor C, the vacuum
3 4
level. The value Z measures the effect of factor D, whether or not the sample is stirred before filling the viscometer. The value
Z measures the effect of factor E, whether the viscometer is vertical or slanted slightly. The value Z measures the effect of factor
6 7
F, the variation of the height of the asphalt when the viscometer is filled. The value Z measures the effect of factor G, the time
that the viscometer is kept in the water bath before testing. Each of these Z-values comprises eight determinations at one level of
the factor and eight determinations at the other level. Therefore, the effect of a factor is obtained by dividing the corresponding
Z-value by eight.
C1067 − 20
TABLE X1.4 Analysis Matrix Based on Applying Signs in Table 1 to Data for Laboratory 1 and Material 1
NOTE 1—The data in Tables X1.5-X1.16 are derived from matrices constructed, as illustrated by this table table, from the data for each of the remaining
eleven laboratory-material combinations from shown in Table X1.3.
Replicate 1 Replicate 2
Row d d d d d d d d d d d d d d d d
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1 2370 2258 2355 2185 1825 1845 1820 1830 2320 2275 2350 2380 1840 1850 1825 1820
2 -2370 -2258 -2355 -2185 1825 1845 1820 1830-2320 -2275 -2350 -2380 1840 1850 1825 1820
2 –2370 –2258 –2355 –2185 1825 1845 1820 1830 –2320 –2275 –2350 –2380 1840 1850 1825 1820
3 -2370 -2258 2355 2185 -1825 -1845 1820 1830-2320 -2275 2350 2380 -1840 -1850 1825 1820
3 –2370 –2258 2355 2185 –1825 –1845 1820 1830 –2320 –2275 2350 2380 –1840 –1850 1825 1820
4 -2370 2258 -2355 2185 -1825 1845 -1820 1830-2320 2275 -2350 2380 -1840 1850 -1825 1820
4 –2370 2258 –2355 2185 –1825 1845 –1820 1830 –2320 2275 –2350 2380 –1840 1850 –1825 1820
5 2370 2258 -2355 -2185 -1825 -1845 1820 1830 2320 2275 -2350 -2380 -1840 -1850 1825 1820
5 2370 2258 –2355 –2185 –1825 –1845 1820 1830 2320 2275 –2350 –2380 –1840 –1850 1825 1820
6 2370 -2258 2355 -2185 -1825 1845 -1820 1830 2320 -2275 2350 -2380 -1840 1850 -1825 1820
6 2370 –2258 2355 –2185 –1825 1845 –1820 1830 2320 –2275 2350 –2380 –1840 1850 –1825 1820
7 2370 -2258 -2355 2185 1825 -1845 -1820 1830 2320 -2275 -2350 2380 1840 -1850 -1825 1820
7 2370 –2258 –2355 2185 1825 –1845 –1820 1830 2320 –2275 –2350 2380 1840 –1850 –1825 1820
8 -2370 2258 2355 -2185 1825 -1845 -1820 1830-2320 2275 2350 -2380 1840 -1850 -1825 1820
8 –2370 –2258 2355 –2185 1825 –1845 –1820 1830 –2320 2275 2350 –2380 1840 –1850 –1825 1820
9 2370 2258 2355 2185 1825 1845 1820 1830-2320 -2275 -2350 -2380 -1840 -1850 -1825 -1820
9 2370 2258 2355 2185 1825 1845 1820 1830 –2320 –2275 –2350 –2380 –1840 –1850 –1825 –1820
10 -2370 -2258 -2355 -2185 1825 1845 1820 1830 2320 2275 2350 2380 -1840 -1850 -1825 -1820
10 –2370 –2258 –2355 –2185 1825 1845 1820 1830 2320 2275 2350 2380 –1840 –1850 –1825 –1820
11 -2370 -2258 2355 2185 -1825 -1845 1820 1830 2320 2275 -2350 -2380 1840 1850 -1825 -1820
11 –2370 –2258 2355 2185 –1825 –1845 1820 1830 2320 2275 –2350 –2380 1840 1850 –1825 –1820
12 -2370 2258 -2355 2185 -1825 1845 -1820 1830 2320 -2275 2350 -2380 1840 -1850 1825 -1820
12 –2370 2258 –2355 2185 –1825 1845 –1820 1830 2320 –2275 2350 –2380 1840 –1850 1825 –1820
13 2370 2258 -2355 -2185 -1825 -1845 1820 1830-2320 -2275 2350 2380 1840 1850 -1825 -1820
13 2370 2258 –2355 –2185 –1825 –1845 1820 1830 –2320 –2275 2350 2380 1840 1850 –1825 –1820
14 2370 -2258 2355 -2185 -1825 1845 -1820 1830-2320 2275 -2350 2380 1840 -1850 1825 -1820
14 2370 –2258 2355 –2185 –1825 1845 –1820 1830 –2320 2275 –2350 2380 1840 –1850 1825 –1820
15 2370 -2258 -2355 2185 1825 -1845 -1820 1830-2320 2275 2350 -2380 -1840 1850 1825 -1820
15 2370 –2258 –2355 2185 1825 –1845 –1820 1830 –2320 2275 2350 –2380 –1840 1850 1825 –1820
16 -2370 2258 2355 -2185 1825 -1845 -1820 1830 2320 -2275 -2350 2380 -1840 1850 1825 -1820
16 –2370 2258 2355 –2185 1825 –1845 –1820 1830 2320 –2275 –2350 2380 –1840 1850 1825 –1820
TABLE X1.5 Results of Calculations for Laboratory 1 and
Material 1
Z = 33148 W = 68,674,369.00 Avg. = 2071.8
1 1
A
Z = -3838 W = 920,640.25 F = 357.41
2 2 A
A
Z = –3838 W = 920,640.25 F = 357.41
2 2 A
Z = -18 W = 20.25 F = 0.01
3 3 B
Z = –18 W = 20.25 F = 0.01
3 3 B
Z = -262 W = 4,290.25 F = 1.67
4 4 C
Z = –262 W = 4,290.25 F = 1.67
4 4 C
Z = -112 W = 784.00 F = 0.30
5 5 D
Z = –112 W = 784.00 F = 0.30
5 5 D
Z = 332 W = 6,889.00 F = 2.67
6 6 E
Z = -8 W = 4.00 F = 0.00
7 7 F
Z = –8 W = 4.00 F = 0.00
7 7 F
Z = -42 W = 110.25 F = 0.04
8 8 G
Z = –42 W = 110.25 F = 0.04
8 8 G
Z = -172 W = 1,849.00
9 9
Z = –172 W = 1,849.00
9 9
Z = 142 W = 1,260.25 s = 2575.88
10 10
Z = -198 W = 2,450.25 s = 50.75
11 11
Z = –198 W = 2,450.25 s = 50.75
11 11
Z = -242 W = 3,660.25
12 12
Z = –242 W = 3,660.25
12 12
Z = 248 W = 3,844.00
13 13
Z = 292 W = 5,329.00
14 14
Z = -128 W = 1,024.00
15 15
Z = –128 W = 1,024.00
15 15
Z = 138 W = 1,190.25
16 16
A
Bold numbers in Tables X1.5-X1.16 indicate statistically significant values.
X1.7 The next step in the analysis is to square the Z-values and divide the squares by 16. The resulting values, which are denoted
W to W are various kinds of “mean sum of squares.” As far as the ruggedness evaluation is concerned, the values W to W are
1 16 2 8
measures of the variance of the means associated with each factor level. For example W is the variance associated with the average
C1067 − 20
values of the determinations obtained at the high and low temperatures. See Appendix X2 for more discussion on the meaning of
the W-values.
X1.8 To determine if a factor has a statistically significant effect, the values of W to W are compared with the error variance (also
2 8
called the mean square error). The error variance is the within-testsingle-operator variance calculated from the replicate
determinations for each of the eight conditions and it indicates the random error associated with the test method. The error variance
is obtained by calculating the sum of W to W and diving by 8 as indicated by Eq 3 in the body of this practice. The calculated
9 16
values of s for each laboratory-material combination are shown in Tables X1.5-X1.16. If there are duplicate determinations, as
is the case in this program, the error variance can also be determined as follows:
k
Δ
(
s 5 (X1.1)
2k
where:
s = error variance or the pooled within-test variance,
s = error variance or the pooled single-operator variance,
Δ = the difference between duplicate determinations, and
k = number of pairs of determinations (k=8 in this program).
X1.9 The final step in the analysis is to compute the F-values for each of the factors by dividing W to W by s as indicated by
2 8
Eq 4 in the body of the practice. The calculated F-values for each laboratory-material combination are shown in Tables
X1.5-X1.16. These values are compared with the critical F-value at a significance level of 0.05 for 1 degree of freedom for the
numerator and 8 degrees of freedom for the denominator. The critical value is 5.32. If the calculated F-value for a factor is ≥5.32,
the factor has a staticallystatistically significant effect with no more than a 5 % probability of making the incorrect inference.
X1.10 The calculated F-values that exceed the critical value are shown as bold numbers in Tables X1.5-X1.16. Table X1.17
summarizes the calculated F-values for all factors anand all laboratory-material combinations. All F-values that are less than 5.32
are indicated in the table as NS to show that they are not statistically significant, and the corresponding factor does not have a
statistically significant effect on the results. The effect of temperature (factor A) was found to be highly significant for every
material and every laboratory indicating the importance of tight control of temperature. The effect of variation in the level of
vacuum (factor C) showed five statistically significant F-values indicating a need for tight tolerance on the applied vacuum. The
TABLE X1.6 Results of Calculations for Laboratory 1 and
Material 2
Z = 7234 W = 3,270,672.25 Avg. = 452.1
1 1
Z = -834 W = 43,472.25 F = 172.51
2 2 A
Z = –834 W = 43,472.25 F = 172.51
2 2 A
Z = -18 W = 20.25 F = 0.08
3 3 B
Z = –18 W = 20.25 F = 0.08
3 3 B
Z = 10 W = 6.25 F = 0.02
4 4 C
Z = 6 W = 2.25 F = 0.01
5 5 D
Z = 26 W = 42.25 F = 0.17
6 6 E
Z = -30 W = 56.25 F = 0.22
7 7 F
Z = –30 W = 56.25 F = 0.22
7 7 F
Z = -18 W = 20.25 F = 0.08
8 8 G
Z = –18 W = 20.25 F = 0.08
8 8 G
Z = 4 W = 1.00
9 9
Z = 16 W = 16.00 s = 252.00
10 10
Z = -24 W = 36.00 s = 15.87
11 11
Z = –24 W = 36.00 s = 15.87
11 11
Z = -124 W = 961.00
12 12
Z = –124 W = 961.00
12 12
Z = 16 W = 16.00
13 13
Z = 116 W = 841.00
14 14
Z = 4 W = 1.00
15 15
Z = 48 W = 144.00
16 16
C1067 − 20
effect of the viscometer deviating from the vertical position (factor E) was statistically significant in six of the laboratory-material
combinations indicating the need for tight tolerance on the alignment of the viscometer. The other factors showed a scattering of
barely significant values, but these were not judged to be of sufficient importance to require tighter controls.
X1.11 Representatives of the three laboratories met after completion of the ruggedness evaluation. After discussion of the results,
the decision was made that it was practicalpracticable and desirable to control temperature, vacuum, and the angle of the viscosity
tube to within the following limits:
Temperature: 25.0 ± 0.1°C
Vacuum: 300 ± 2 mmHg
Angle with horizontal: 90.0 ± 1.0°
C1067 − 20
X1.11.1 With these changes, an interlaboratory study was organized and carried out using the revised test method.
TABLE X1.7 Results of Calculations for Laboratory 1 and
Material 3
Z = 58618 W = 214,754,370.25 Avg. = 3663.6
1 1
Z = -6898 W = 2,973,900.25 F = 586.74
2 2 A
Z = –6898 W = 2,973,900.25 F = 586.74
2 2 A
Z = -312 W = 6,084.00 F = 1.20
3 3 B
Z = –312 W = 6,084.00 F = 1.20
3 3 B
Z = -624 W = 24,336.00 F = 4.80
4 4 C
Z = –624 W = 24,336.00 F = 4.80
4 4 C
Z = 456 W = 12,996.00 F = 2.56
5 5 D
Z = 764 W = 36,481.00 F = 7.20
6 6 E
Z = -214 W = 2,862.25 F = 0.56
7 7 F
Z = –214 W = 2,862.25 F = 0.56
7 7 F
Z = 218 W = 2,970.25 F = 0.59
8 8 G
Z = -318 W = 6,320.25
9 9
Z = –318 W = 6,320.25
9 9
Z = 206 W = 2,652.25 s = 5068.50
10 10
Z = -216 W = 2,916.00 s = 71.19
11 11
Z = –216 W = 2,916.00 s = 71.19
11 11
Z = -248 W = 3,844.00
12 12
Z = –248 W = 3,844.00
12 12
Z = -288 W = 5,184.00
13 13
Z = –288 W = 5,184.00
13 13
Z = 540 W = 18,225.00
14 14
Z = -150 W = 1,406.25
15 15
Z = –150 W = 1,406.25
15 15
Z = 2 W = 0.25
16 16
TABLE X1.8 Results of Calculations for Laboratory 1 and
Material 4
Z = 14692 W = 13,490,929.00 Avg. = 918.3
1 1
Z = -1892 W = 223,729.00 F = 828.24
2 2 A
Z = –1892 W = 223,729.00 F = 828.24
2 2 A
Z = -208 W = 2,704.00 F = 10.01
3 3 B
Z = –208 W = 2,704.00 F = 10.01
3 3 B
Z = -122 W = 930.25 F = 3.44
4 4 C
Z = –122 W = 930.25 F = 3.44
4 4 C
Z = 232 W = 3,364.00 F = 12.45
5 5 D
Z = 94 W = 552.25 F = 2.04
6 6 E
Z = -78 W = 380.25 F = 1.41
7 7 F
Z = –78 W = 380.25 F = 1.41
7 7 F
Z = 162 W = 1,640.25 F = 6.07
8 8 G
Z = 26 W = 42.25
9 9
Z = -18 W = 20.25 s = 270.13
10 10
Z = –18 W = 20.25 s = 270.13
10 10
Z = -2 W = 0.25 s = 16.44
11 11
Z = –2 W = 0.25 s = 16.44
11 11
Z = -116 W = 841.00
12 12
Z = –116 W = 841.00
12 12
Z = 18 W = 20.25
13 13
Z = 120 W = 900.00
14 14
Z = -64 W = 256.00
15 15
Z = –64 W = 256.00
15 15
Z = 36 W = 81.00
16 16
C1067 − 20
TABLE X1.9 Results of Calculations for Laboratory 2 and
Material 1
Z = 32692 W = 66,797,929.00 Avg. = 2043.3
1 1
Z = -3708 W = 859,329.00 F = 813.76
2 2 A
Z = –3708 W = 859,329.00 F = 813.76
2 2 A
Z = -190 W = 2,256.25 F = 2.14
3 3 B
Z = –190 W = 2,256.25 F = 2.14
3 3 B
Z = -516 W = 16,641.00 F = 15.76
4 4 C
Z = –516 W = 16,641.00 F = 15.76
4 4 C
Z = 130 W = 1,056.25 F = 1.00
5 5 D
Z = 544 W = 18,496.00 F = 17.52
6 6 E
Z = -358 W = 8,010.25 F = 7.59
7 7 F
Z = –358 W = 8,010.25 F = 7.59
7 7 F
Z = 382 W = 9,120.25 F = 8.64
8 8 G
Z = -32 W = 64.00
9 9
Z = –32 W = 64.00
9 9
Z = 8 W = 4.00 s = 1056.00
10 10
Z = -30 W = 56.25 s = 32.50
11 11
Z = –30 W = 56.25 s = 32.50
11 11
Z = 16 W = 16.00
12 12
Z = 10 W = 6.25
13 13
Z = 76 W = 361.00
14 14
Z = 218 W = 2,970.25
15 15
Z = -282 W = 4,970.25
16 16
Z = –282 W = 4,970.25
16 16
TABLE X1.10 Results of Calculations for Laboratory 2 and
Material 2
Z = 7543 W = 3,556,053.06 Avg. = 471.4
1 1
Z = -803 W = 40,300.56 F = 331.86
2 2 A
Z = –803 W = 40,300.56 F = 331.86
2 2 A
Z = -53 W = 175.56 F = 1.45
3 3 B
Z = –53 W = 175.56 F = 1.45
3 3 B
Z = -57 W = 203.06 F = 1.67
4 4 C
Z = –57 W = 203.06 F = 1.67
4 4 C
Z = 73 W = 333.06 F = 2.74
5 5 D
Z = 81 W = 410.06 F = 3.38
6 6 E
Z = -97 W = 588.06 F = 4.84
7 7 F
Z = –97 W = 588.06 F = 4.84
7 7 F
Z = 49 W = 150.06 F = 1.24
8 8 G
Z = 15 W = 14.06
9 9
Z = -15 W = 14.06 s = 121.44
10 10
Z = –15 W = 14.06 s = 121.44
10 10
Z = 11 W = 7.56 s = 11.02
11 11
Z = -57 W = 203.06
12 12
Z = –57 W = 203.06
12 12
Z = -51 W = 162.56
13 13
Z = –51 W = 162.56
13 13
Z = 61 W = 232.56
14 14
Z = 71 W = 315.06
15 15
Z = -19 W = 22.56
16 16
Z = –19 W = 22.56
16 16
X2. THEORY OF THE RUGGEDNESS ANALYSIS
X2.1 Introduction
X2.1.1 Any statistical analysis depends on assumptions. Because a ruggedness or screening program is usually run on a new test
method, there is little history or experience to validate the necessary assumptions. An extensive study could yield the experience
to validate the assumptions, but it would also increase the cost of the ruggedness program to the point that few such programs could
be undertaken. This practice seeks to balance these risks by making plausible assumptions to make the practice practicalpracticable
and useful.
C1067 − 20
TABLE X1.11 Results of Calculations for Laboratory 2 and
Material 3
Z = 58527.00 W = 214,088,108.06 Avg. = 3657.9
1 1
Z = -7123.00 W = 3,171,070.56 F = 226.64
2 2 A
Z = –7123.00 W = 3,171,070.56 F = 226.64
2 2 A
Z = -755.00 W = 35,626.56 F = 2.55
3 3 B
Z = –755.00 W = 35,626.56 F = 2.55
3 3 B
Z = -423.00 W = 11,183.06 F = 0.80
4 4 C
Z = –423.00 W = 11,183.06 F = 0.80
4 4 C
Z = 247.00 W = 3,813.06 F = 0.27
5 5 D
Z = 191.00 W = 2,280.06 F = 0.16
6 6 E
Z = 443.00 W = 12,265.56 F = 0.88
7 7 F
Z = -171.00 W = 1,827.56 F = 0.13
8 8 G
Z = –171.00 W = 1,827.56 F = 0.13
8 8 G
Z = -145.00 W = 1,314.06
9 9
Z = –145.00 W = 1,314.06
9 9
Z = 433.00 W = 11,718.06 s = 13991.81
10 10
Z = -171.00 W = 1,827.56 s = 118.29
11 11
Z = –171.00 W = 1,827.56 s = 118.29
11 11
Z = -55.00 W = 189.06
12 12
Z = –55.00 W = 189.06
12 12
Z = -77.00 W = 370.56
13 13
Z = –77.00 W = 370.56
13 13
Z = 1107.00 W = 76,590.56
14 14
Z = -413.00 W = 10,660.56
15 15
Z = –413.00 W = 10,660.56
15 15
Z = 385.00 W = 9,264.06
16 16
TABLE X1.12 Results of Calculations for Laboratory 2 and
Material 4
Z = 15095 W = 14,241,189.06 Avg. = 943.4
1 1
Z = -1969 W = 242,310.06 F = 269.21
2 2 A
Z = –1969 W = 242,310.06 F = 269.21
2 2 A
Z = -179 W = 2,002.56 F = 2.22
3 3 B
Z = –179 W = 2,002.56 F = 2.22
3 3 B
Z = -149 W = 1,387.56 F = 1.54
4 4 C
Z = –149 W = 1,387.56 F = 1.54
4 4 C
Z = 265 W = 4,389.06 F = 4.88
5 5 D
Z = 135 W = 1,139.06 F = 1.27
6 6 E
Z = 5 W = 1.56 F = 0.00
7 7 F
Z = 101 W = 637.56 F = 0.71
8 8 G
Z = -115 W = 826.56
9 9
Z = –115 W = 826.56
9 9
Z = 121 W = 915.06 s = 900.06
10 10
Z = 67 W = 280.56 s = 30.00
11 11
Z = -195 W = 2,376.56
12 12
Z = –195 W = 2,376.56
12 12
Z = -69 W = 297.56
13 13
Z = –69 W = 297.56
13 13
Z = 189 W = 2,232.56
14 14
Z = -65 W = 264.06
15 15
Z = –65 W = 264.06
15 15
Z = 11 W = 7.56
16 16
X2.1.2 A ruggedness program attempts to identify the important factors that cause variability of results obtained using the test
method. It is important that all of the major factors be included in the study, because if one is left out, the study will not be able
to identify its significance. The procedure in this practice is set up to evaluate the effects of seven factors using as few tests as
possible. This is usually sufficient to cover the major sources of variability. Designs Experimental designs for both fewer and more
factors are given in statistical texts for use when needed (1, 2, 3).
X2.1.3 It is unusual to need to investigate more than seven factors and it is typical to have at least five factors that can be varied.
WhenIf only five factors are considered to be potentially significant, two other factors can nearly always be selected about which
there may be some doubt. A seven factor analysis is usually suitable for most screening programs to evaluate the ruggedness of
a test method.
The boldface numbers in parentheses refer to a list of references at the end of this standard.
C1067 − 20
TABLE X1.13 Results of Calculations for La
...