ASTM E3327/E3327M-21

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation:E3327/E3327M −21
Standard Guide for
the Qualification and Control of the Assisted Defect
Recognition of Digital Radiographic Test Data
ThisstandardisissuedunderthefixeddesignationE3327/E3327M;thenumberimmediatelyfollowingthedesignationindicatestheyear
of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope priate for other nondestructive test methods when approved by
the AssistDR system purchaser.
1.1 Assisted defect recognition (AssistDR) describes a class
of computer algorithms that assist a human operator in making 1.5 Units—The values stated in either SI units or inch-
a determination about nondestructive test data. This guide uses pound units are to be regarded separately as standard. The
the term AssistDR to describe those computer assisted evalu- values stated in each AssistDR system may not be exact
ation algorithms and associated software. For the purposes of equivalents; therefore, each AssistDR system should be used
this guide, the usage of the words “defect,” “evaluate,” independently of the other.
“evaluation,” etc., in no way implies that the algorithms are
1.6 This standard does not purport to address all of the
dispositioning or otherwise making an unaided final disposi-
safety concerns, if any, associated with its use. It is the
tion. Depending on the application, AssistDR computer algo-
responsibility of the user of this standard to establish appro-
rithms detect and optionally classify indications of defects,
priate safety, health, and environmental practices and deter-
flaws, discontinuities, or other anomalous signals in the ac-
mine the applicability of regulatory limitations prior to use.
quired images. Software that does make an unaided final
1.7 This international standard was developed in accor-
disposition is classified as automated defect recognition (Au-
dance with internationally recognized principles on standard-
toDR).While the concepts discussed in this guide are pertinent
ization established in the Decision on Principles for the
to AutoDR applications, additional validation tests or controls
Development of International Standards, Guides and Recom-
may be necessary when implementing AutoDR.
mendations issued by the World Trade Organization Technical
Barriers to Trade (TBT) Committee.
1.2 This guide establishes the minimum considerations for
the radiographical examination of components usingAssistDR
2. Referenced Documents
for non-film radiographic test data. Most of the examples and
discussion in this guide are built around two-dimensional test
2.1 ASTM Standards:
data for simplicity. The principles can be applied to three
E1316 Terminology for Nondestructive Examinations
(volumetric computed tomography, for example) or higher
E1441 Guide for Computed Tomography (CT)
dimensional test data.
E1695 Test Method for Measurement of Computed Tomog-
raphy (CT) System Performance
1.3 The methods and practices described in this guide are
E2033 Practice for Radiographic Examination Using Com-
intended for the application ofAssistDR where image analysis
puted Radiography (Photostimulable Luminescence
will aid a human operator in the detection and evaluation of
Method)
indications. The degree to which AssistDR is integrated into
E2339 Practice for Digital Imaging and Communication in
the testing and evaluation process will help the user determine
Nondestructive Evaluation (DICONDE)
the appropriate levels of process qualification and control
E2422 Digital Reference Images for Inspection of Alumi-
required.This guide is not intended for applications wishing to
num Castings
employ AutoDR in which there is no human review of the
E2445/E2445M Practice for Performance Evaluation and
results.
Long-Term Stability of Computed Radiography Systems
1.4 This guide applies to radiographic examination using an
E2586 Practice for Calculating and Using Basic Statistics
X-ray source. Some of the concepts presented may be appro-
E2597/E2597M PracticeforManufacturingCharacterization
of Digital Detector Arrays
This guide is under the jurisdiction of ASTM Committee E07 on Nondestruc-
tive Testing and is the direct responsibility of Subcommittee E07.01 on Radiology For referenced ASTM standards, visit the ASTM website, www.astm.org, or
(X and Gamma) Method. contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
Current edition approved Dec. 1, 2021. Published February 2022. DOI: 10.1520/ Standards volume information, refer to the standard’s Document Summary page on
E3327_E3327M-21. the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3327/E3327M−21
E2698 Practice for Radiographic Examination Using Digital 3.2.6 failure mode and effects analysis (FMEA), n—the
Detector Arrays systematic process of reviewing as many components and
E2862 Practice for Probability of Detection Analysis for subsystems as possible to identify potential risks in a system
Hit/Miss Data and their root causes and impact.
E2737 Practice for Digital Detector Array Performance
3.2.7 false negative (FN), n—an examination result that
Evaluation and Long-Term Stability
reports that no indication is present when there is an indication
E3023 Practice for Probability of Detection Analysis for â
in the ground truth, that is, a missed identification of an
Versus a Data
indication that should have been detected, sometimes called a
E3169 Guide for Digital Imaging and Communication in
“miss.”
Nondestructive Evaluation (DICONDE)
3.2.8 false positive (FP), n—an examination result that
2.2 ISO Standards:
reports that an indication is present when there is no corre-
ISO 9000 Family Quality Management
spondingindicationinthegroundtruth,thatis,identificationof
2.3 Other Documents: an indication that should not have been identified, sometimes
called a “false call.”
NEMA PS3 / ISO 12052Digital Imaging and Communica-
tions in Medicine (DICOM) Standard, National Electrical
3.2.9 false positive rate (FPR), n—number of false positive
Manufacturers Association, Rosslyn, VA, USA (available
examination results divided by the total opportunities.
free at http://www.dicomstandard.org/)
3.2.10 ground truth, n—list of indications and associated
MIL-HDBK-1823A Nondestructive Evaluation System Re-
metadata present in the test data as determined by the process
liability Assessment
expert.
3.2.11 negative, n—an examination result that does not
3. Terminology
report the presence of an indication.
3.1 See Terminology E1316 as well as the ASTM CT, CR,
3.2.12 negative predictive value (NPV), n—the probability
and DDA standards listed in Section 2 for a complete set of
that a negative is a true negative.
standard non-film radiographic test method definitions.
3.2.13 opportunity, n—a single occurrence of the unit of
3.2 Definitions of Terms Specific to This Standard:
measure for the examination, for example, a part, an image, or
3.2.1 assisted defect recognition (AssistDR), n—the soft-
a pixel.
ware or computer algorithms, typically involving image
segmentation, feature identification, classification, and 3.2.14 positive, n—an examination result that reports the
measurement, that aid operators in detecting and optionally the
presence of an indication.
evaluation of indications in digital nondestructive testing data.
3.2.15 positive predictive value (PPV), n—the probability
Also referred to as software assisted evaluation, computer
that a positive is a true positive.
assisted evaluation, computer assisted detection, semi-
3.2.16 probability of detection (POD), n—a method to
automated defect recognition, supervised automated defect
quantitatively assess the performance of a test method de-
recognition, or computer aided detection.
scribed in Practice E2862, Practice E3023, and MIL-HDBK-
3.2.2 automated defect recognition (AutoDR), n—the soft-
1823A.
ware or computer algorithms, typically involving image
3.2.17 process expert, n—the individual or group of indi-
segmentation, feature identification, classification, and
viduals responsible for establishing ground truth for the data
measurement, that classify the part being tested as acceptable
used in the examination, for example, subject matter experts,
or rejectable without the involvement of an operator.
AssistDR system experts, certified Level 3s, statisticians, etc.
3.2.3 confidence interval, n—a range of values such that
3.2.18 receiver operator characteristic curve (ROC), n—a
there is a specified probability that the value of an unknown
graphical plot that illustrates the diagnostic ability of a binary
constant parameter of interest is contained in the range.
classifier system as its discrimination threshold is varied. The
3.2.4 confidence level, n—the probability that a specified
ROC curve is created by plotting the true positive rate (TPR)
range of values covers an unknown constant parameter of
against the false positive rate (FPR) at various threshold
interest.
settings.
3.2.5 data curation, v—the organization and integration of
3.2.19 sensitivity, n—measure of the portion of true posi-
data collected from various sources, involving annotation,
tives that are correctly identified in an examination. Sensitivity
publication, and presentation of the data such that the value of
is synonymous with true positive rate (TPR).
thedataismaintainedovertime,andthedataremainsavailable
3.2.20 specificity, n—measure of the portion of true nega-
for reuse and preservation.
tives which are correctly identified in an examination. Speci-
ficity is synonymous with true negative rate (TNR).
3.2.21 total indications, n—the number of indications pres-
Available from International Organization for Standardization (ISO), ISO
Central Secretariat, Chemin de Blandonnet 8, CP 401, 1214 Vernier, Geneva,
ent in the ground truth.
Switzerland, https://www.iso.org.
3.2.22 total opportunities, n—thenumberofpossibilitiesfor
Available from http://everyspec.com/MIL-HDBK/MIL-HDBK-1800-1999/
MIL-HDBK-1823A_33187/. indications to be identified in the ground truth.
E3327/E3327M−21
3.2.23 true negative (TN), n—an examination result that 4.3 Initial Considerations—Several often-overlooked items
reports that no indication is present when there is no corre- should be considered before undertaking a project to imple-
sponding indication in the ground truth. ment an AssistDR process. Are the image chain and X-ray
technique optimized for software evaluation? Is a statistically
3.2.24 true negative rate (TNR), n—the number of true
significant amount of the data available from the inspection
negative examination results divided by the total opportunities.
process? If so, is curated ground truth available for that data,
3.2.25 true positive (TP), n—an indication in the examina-
and is that data representative of all indication types? Are the
tion results that corresponds to an indication in the ground
target part manufacturing and inspection processes mature
truth, that is, identification of an indication that should have
enough to execute AssistDR development without repeated
been identified, sometimes called a “hit.”
requalification?
3.2.26 true positive rate (TPR), n—the number of true
4.4 Data Collection—Significant amounts of data both with
positive examination results divided by the total indications.
and without indications is needed to have a successful imple-
3.2.27 type I error, n—the incorrect rejection of a true null
mentation of AssistDR. Hundreds, if not thousands, of images
hypothesis (a "false positive").
will be needed for both the development/training of the
3.2.28 type II error, n—the incorrect acceptance of a false
software algorithms as well as the initial qualification and
null hypothesis (a "false negative").
future requalifications. This guide describes the different data
types needed for AssistDR and best practices for assembling
3.2.29 yield, n—the percentage of manufactured compo-
that data.
nents that are evaluated as conforming.
4.5 Process Qualification—Once the AssistDR system is
4. Summary of Guide
trained using a set of data for which ground truth has been
4.1 This document is written in sections that correspond to
provided, it should then be qualified without knowledge of the
thestagesinthelifecycleofanAssistDRimplementationfrom
ground truth to understand its real performance.After training,
initialconceptthroughproductionoperationasshowninFig.1.
anexpectedTPRandFPRareknownfromaReceiverOperator
4.1.1 Following the life cycle, first the performance mea-
Characteristic (ROC) chart, but the confidence in those results
surements for the system are defined and agreed upon. Next,
may be low because the training is inherently biased by
data collection, image quality, data availability, and data
knowledge of the ground truth for the data. Therefore, quali-
curation are initiated. Once a curated data set is available,
ficationshouldbeconductedwithastatisticallysignificantdata
process training and qualification for use can occur. Production
set. The size of the qualification data set is determined by the
use of the AssistDR system occurs after qualification is
sample size calculated based on expected TPR and FPR, the
complete. The production use of theAssistDR system needs to
required confidence level, and the required confidence interval
be controlled in a manner similar to other nondestructive
as described in 6.3. It should be noted that the qualification
testing processes. At some point during the life cycle, the
database should incorporate the same types of indications from
AssistDR system will need to be maintained or the manufac-
the FMEA as the training database, supplementing with syn-
turing process will change for the part being inspected. When
thetic data where necessary. Also, any test data that results in
AssistDRsystemmaintenanceorupgradeoccurs,orchangesto
the AssistDR system being changed should be added to the
the part manufacturing process occur, the AssistDR system
training database and removed from the qualification database.
performanceneedstobeverified.Ifthechangestothesoftware
The process of qualification is shown in Fig. 2.
or process are significant enough, the system may need to be
4.5.1 The operating point on the ROC chart generated at the
requalified. A summary of each of the steps in the life cycle
qualification phase therefore should be the reportable TPR and
follows below.
FPR for the AssistDR process, and these values should have
4.2 Performance Measurement—In order to determine if an confidence intervals associated with them that meet the re-
inspection process utilizingAssistDR is equivalent to or better quirements of the application. For instance, some applications
than the existing inspection process, the performance of the that utilize AssistDR only as a tool to call attention to an
existing inspection process needs to be understood. Common abnormal condition would require a significantly different
definitions for inspection system performance metrics and operating point than an application where AssistDR is used as
methods for measuring those metrics for both operator and required or critical input to the operator’s decision. Once
AssistDR are described in this guide. qualified, all test data that has influenced AssistDR system
FIG. 1Overview of an AssistDR System Life Cycle
E3327/E3327M−21
FIG. 2Qualification Process for AssistDR
performance has been incorporated into the training database and uncertainty present in any inspection process, acceptance
and removed from the qualification database. The qualification thresholds are established so that some acceptable components
database can serve as a basis for the regression and requalifi- are discarded in an effort to prevent parts with discontinuities
cation test databases for future requalification when AssistDR that exceed the acceptance criteria from entering service. This
system changes or improvements occur. type of error, called a false positive, is considered less critical
thanafalsenegativeerrorwhichwouldallowanonconforming
4.6 Process Control—Once an AssistDR process has been
part into service. A successful application of AssistDR mini-
qualified for use in production, a method for monitoring and
mizes the false positive rate while reducing the false negative
controlling the performance of the process will be needed.
rate to levels appropriate for the intended application. The
Strategies for both the monitoring and control of an inspection
methods and techniques described in this guide facilitate
process using AssistDR are presented in this guide.
achieving this desired outcome.
4.7 Process Maintenance—Similar to other inspection
5.4 With the advent of deep learning, convolutional neural
processes, both routine and special cause maintenance occur
networks, and other forms of artificial intelligence, scenarios
for AssistDR processes. Equipment or software upgrades or
become possible where an AssistDR system continues to
replacements may also occur. When these events occur, the
evolve or learn after qualification for production use. This
potential impact of those events on the performance of the
guide does not address learning-basedAssistDR systems. This
system needs to be assessed. Recommendations for assessing
guide addresses only deterministic systems that have software
and measuring the performance impact of maintenance events
code and parameters that are fixed after qualification. Note that
are detailed in this guide.
this limitation does not prohibit the use of this guide for
developing a qualification and usage strategy for software
5. Significance and Use
using deep learning technology. The training or learning
5.1 This guide describes the recommended procedure for
process for the deep learning system would need to be
using software to assist with the identification of indications in
completed before qualification and all parameters of the deep
digital radiographic images. Some of the concepts presented
learning system held fixed (as with deterministic software
may be appropriate for other nondestructive test methods.
approaches based on traditional image processing) after quali-
5.2 When properly applied, the methods and techniques
fication and during use.
outlined in this guide offer radiographic testing practitioners
the potential to improve inspection reliability, reduce inspec- 6. Performance Measurement
tion cycle time, and harness inspection statistics for improving
6.1 The ability of an AssistDR system to find relevant
manufacturing processes.
indications in the nondestructive test data and to ignore
5.3 The typical goal of a nondestructive test is to identify nonrelevant ones is the cornerstone of a successful implemen-
flaws that exceed the acceptance criteria. Due to the variability tation. This section defines common performance measures for
E3327/E3327M−21
AssistDR processes. Since those measures will be calculated 6.3.1 Thetruevalueoftheperformancemetricsdescribedin
from a sample of the processed data, the relevant statistical 6.2 is never completely known. The performance of operators
measuresontheconfidenceofthosemetricsaredescribednext. or AssistDR systems can never be measured on every part
An AssistDR project will need to set targets for both the produced.Instead,theperformancemetricsareestimatedusing
performance measures and statistical confidence during initial
a sample of parts. This single measurement does not necessar-
phases. Finally, some guidance on determining the sample size ily reflect the true performance, but its statistical confidence
needed to meet those targets is presented. The AssistDR
can also be measured and reported. Due to this measurement
process should be evaluated under the same conditions that it uncertainty, the statistical significance of a sample is expressed
is intended to be used. For example, when measuring the
using a confidence interval and confidence level. This allows
performance of AssistDR, it should include operator input if
for the precision of the performance measurement to be
that is the intended production use.
reported.
6.3.2 The interpretation of confidence interval and confi-
6.2 Quantitative System Performance Evaluation
dence level on TPR is shown in Fig. 4. The endpoints of the
6.2.1 The first step in analyzing AssistDR system perfor-
confidence interval are referred to as the upper and lower
mance is to organize its results into the format shown in Fig. 3.
confidence bounds. For simplicity, Fig. 4 shows a special case
In this table, the number of positive examination results that
had a corresponding indication in the ground truth is put in the of upper and lower confidence bounds that are symmetrically
first row and first column and the number of positive exami- distributed about the sample estimate. Because the distribution
nation results that did not have a corresponding indication in is symmetric, the confidence level is divided into two equal
the ground truth goes in the second row of column 1.The same halves to create both the upper and lower confidence bounds.
process is followed for the second column of the table for the For a TPR estimate t with confidence bounds cb and cb, and
u l
negative examination results. A table in this format is often a confidence level c%, the probability that the sample’s upper
referred to as a truth table or confusion matrix. and lower bounds, cb and cb, contain the population TPR is
u l
6.2.2 The number of total indications for the examination c%.
result in Fig. 3 is a+b.
6.3.3 For the measurement of TPR, the symmetric probabil-
6.2.3 The number of true positive results for the examina-
ity distribution above is appropriate only forTPRs measured at
tion in Fig. 3 is a. The true positive rate can be calculated as a
or near 50 %, or for very large sample sizes. As the TPR
/ (a+b).
approaches 100 %, the distribution skews left and sample TPR
6.2.4 The number of false negative results for the examina-
does not coincide with the mode or peak of the distribution.
tion in Fig. 3 is b. These are Type II errors. The false negative
Additionally, neither the confidence level nor the confidence
rate can be calculated as b / (a+b).
intervals are symmetric. The width of the confidence interval
6.2.5 Thenumberofopportunitiesforafalsepositiveforthe
dependsonthesamplesize,N.Fig.5showsthisrelationshipas
examination result in Fig. 3 is c+d.
TPR deviates from the simple case, for a relatively low sample
6.2.6 The number of false positive results for the examina-
size of 30. For more information regarding skewness of
tion in Fig. 3 is c. These are Type I errors. The false positive
probability density functions and appropriate probability dis-
rate can be calculated as c / (c+d).
tribution models for TPR and FPR estimates, refer to Practice
6.2.7 The number of true negative results for the examina-
E2586.
tion in Fig. 3 is d. The true negative rate can be calculated as
6.3.4 It is the lower confidence bound that is of interest for
d / (c+d).
the measurement of the TPR of an AssistDR application. The
6.2.8 The fraction of examinations that correctly tested
lower confidence bound on the TPR estimate represents the
positive for an indication is the positive predictive value
lowest value ofTPR that could be expected on future measure-
(PPV). The PPV can be calculated as a / (a+c).
ments of that AssistDR process at a given confidence level.
6.2.9 The fraction of examinations that correctly tested
Note that for skewed sensitivity distributions as shown in Fig.
negative for an indication is the negative predictive value
6, the confidence interval is asymmetric including more TPR
(NPV). The NPV can be calculated as d / (b+d).
estimates below the measured sensitivity. Hence, the lower
6.3 Use of Confidence Intervals in AssistDR Validation boundisfurtherbelowthemeasuredTPRthantheupperbound
FIG. 3AssistDR Test Results Compared to Ground Truth
E3327/E3327M−21
FIG. 4Illustration of Confidence Bounds and Confidence Level
FIG. 5Illustration of Confidence Level and Confidence Intervals for Increasing TPR Measurements
is above. Increasing the sample size reduces this effect on the 6.4 Sample Size Requirements for Validation of Initial
sensitivity measurement’s lower bound. Similarly, it is the Performance Assessments
upper confidence bound that is of most interest for FPR 6.4.1 In order to provide a clear, quantitative validation of
estimates. an AssistDR system’s performance (in terms of TPR, for
E3327/E3327M−21
FIG. 695% Confidence Interval Width for TPR Estimates as A Function of Sample Size and TPR; Lines Are Colored by Estimated TPR,
Increasing From 70% (in Black) to 98% (in Red); Note That Confidence Interval Width Decreases as A Function of Sample Size And
That Confidence Interval Widths Are Smaller for Larger TPR Estimates Than for Lower TPR Estimates For A Given Sample Size
example) on a given part, two quantities are required before- set with 250 indications. The row of Table 1 that is closest to,
hand. The first of these quantities is an estimate of the butnotgreaterthan,theTPRisthe90 %TPRrow.Thecolumn
anticipated performance (say TPR) and the second is the with the number of indications that is closest to, but not greater
desired width of the resulting confidence interval. These two than, the 250 indications used in the study is the 5 %
quantities are needed because the relationship between sample ConfidenceIntervalcolumn.The95 %lowerconfidencebound
size and confidence interval width is a function of anticipated on TPR for this study is 85.5 % (90.5 % - 5 %).
performance. That is, the number of samples required to meet 6.4.3 Similarly, an upper confidence bound for an FPR
a given confidence interval width for one value of TPR will estimate can be estimated given the sample size used to
differ from the number required for a higher value ofTPR. Fig. calculatetheestimateandthedesiredconfidenceinterval.Ifthe
6 illustrates this relationship for TPR. numberofopportunitiesforafalsepositiveisdefined,FPRcan
6.4.2 ThelowerconfidenceboundforaTPRestimatecanbe be bounded. By noting that FPR=1– TNR, Table 1 can also
estimated given the sample size used to calculate the estimate be used to compute a lower bound on TNR. Analogously, this
and the desired confidence level. The approximate lower lower bound on TNR equates to an upper bound on FPR.
confidence bound for a 95 % confidence level for a givenTPR 6.4.4 If the number of opportunities for a false positive is
andsamplesizeintheassociatedstudycanbedeterminedfrom not defined, FPR may be unbounded. For example, if it is
Table 1. To use Table 1, first find the row that is closest to, but desired to measure the number of false positives per
not greater than, the TPR of the study. Next, find the column opportunity, the upper confidence bound should be calculated
with the number of samples that is closest to, but not greater from a counting process distribution, as described in Practice
than, the number of samples in the study. The heading of the E2586. The approximate upper confidence bound for a 95 %
column is the lower confidence interval. To determine the confidence level for a given number of false positives in the
lower confidence bound, simply subtract the lower confidence associated study can be determined from Table 2. To use Table
interval from the TPR of the study. For example, consider a 2, first find the row that is closest to, but not less than, the
study with a 90.5 % TPR measured from a qualification data number of false positives per opportunity of the study. Next,
TABLE 1 Number of Indications Required in the Qualification Data Set for a 95% Confidence Level (CL) on True Positive Rate
True Positive Rate Confidence Interval (CI)
1% 2% 3% 4% 5% 10% 20%
97.0 % 1484 461 244 159 115 45 19
96.0 % 1837 548 282 180 129 49 20
95.0 % 2181 632 319 201 142 52 20
94.0 % 2517 715 356 221 155 55 21
93.0 % 2846 796 391 240 167 57 21
92.0 % 3167 875 425 260 179 60 22
91.0 % 3480 952 459 278 191 63 23
90.0 % 3786 1027 492 296 202 65 23
87.5 % 4516 1206 570 339 229 71 24
85.0 % 5198 1373 643 379 254 77 25
82.5 % 5832 1528 710 416 277 82 26
80.0 % 6418 1671 772 450 298 87 27
E3327/E3327M−21
TABLE 2 Number of Opportunities Required in the Qualification Data Set for a 95% Confidence Level (CL) on False Positives per
Opportunity
False Positives Per Confidence Interval (CI)
Opportunity 0.05 0.10 0.20 0.30 0.40 0.50 1.00
0.10 231 77 29 17 12 - -
0.20 385 116 39 22 15 7 -
0.30 539 154 48 26 17 8 -
0.40 693 193 58 30 20 8 6
0.50 846 231 68 35 22 9 6
0.60 1000 270 77 39 24 10 7
0.70 1154 308 87 43 27 10 7
0.80 1307 347 97 47 29 11 7
0.90 1461 385 106 52 32 12 8
1.00 1615 423 116 56 34 12 8
1.25 1999 520 140 67 40 14 9
1.50 2383 616 164 77 46 16 10
1.75 2767 712 188 88 52 18 11
2.00 3151 808 212 99 48 19 12
4.00 6225 1576 404 184 106 33 20
find the column with the number of samples that is closest to, Thisisdescribedindetailin7.2.Thisdefinitionshouldbeused
but not less than, the number of samples in the study. The as guidance to collect data sets for assessing operator
heading of the column is the upper confidence interval. To performance, software training, and software performance. For
determine the upper confidence bound, add the upper confi- thisdatatobeusefulinconductingperformanceassessmentsor
dence interval from the number of false positives per oppor- training, it will need to have the ground truth determined.
tunity of the study. For example, consider a study with 0.83 Ground truth determination is discussed in 7.3.
false positives per clean component measured from a qualifi-
7.1.1 Before describing the elements of a comparison for
cation data set with 100 clean components. The row of Table 2 evaluating an AssistDR system, it should be noted that every
that is closest to, but not less than, the false positives per clean
inspection system and manufacturing context is different, and
component is the 0.9 false positives row. The column with the thosedifferencescanhavesignificantimplicationsforconduct-
number of opportunities that is closest to, but not less than, the
ing a successful and meaningful comparison study. To handle
100 opportunities used in the study is the 0.2 Confidence these nuances and to ensure the accuracy of results, it is
Interval column. The 95 % upper confidence bound on false
recommended that a trained statistician be consulted whenever
positives per component for this study is 1.1 (0.9 + 0.2). possible. If theAssistDR system purchaser or provider lacks a
staff statistician, there are a variety of fully qualified statistical
6.5 An alternative approach to the quantitative assessment
consulting firms that can be contacted to assist in the design of
of the performance of a nondestructive testing method is
these studies.
probability of detection (POD). Like TPR, the POD approach
provides estimates of the probability that an inspection method
7.2 Representative Process Data
correctly detects defects. The POD approach is based upon a
7.2.1 Often, faults in AssistDR systems are not detected
statistical model with several important assumptions. Perhaps
until the later stages of development. The late detection is due
the most important of these assumptions is that there is a single
to the random nature of the variation of manufacturing pro-
aspect of true defects (often defect size) that strongly influ-
cesses. The image or type of indication that caused the fault
ences probability of detection.TPR, on the other hand, rests on
was simply not produced by the process during the previous
very few assumptions. The assumptions behind both methods
testing periods. However, finding a problem at this point in the
have both benefits and risks. The strongly parametric nature of
development process can add significant cost and delays to
the POD approach means that smaller data sets are required to
schedules. The challenge is to create a training data set that is
get meaningful estimates, while the simpler TPR approach
asrobustandcomprehensiveaspossibletouncoverthesefaults
requires larger data sets. However, TPR is ultimately more
during the training phase. One way to create a robust training
robust in the presence of multiple defect types or when image
set is to use Failure Mode and EffectsAnalysis (FMEA), a tool
properties other than indication size have strong influence on
for identifying potential problems and their impact. FMEAis a
performance (for example, strong gradients or variation in
qualitative and systematic tool, usually created within a
contrast). Detailed information on POD can be found in
spreadsheet, to help practitioners anticipate what might go
Practice E2862, Practice E3023, and MIL-HDBK-1823A.
wrong with a product or process. Since the effects of failure for
AssistDR systems are limited, the focus is on identifying
7. Data Collection
modes (indication types, indication locations, background
level,backgroundgradient,etc.)thatmaycausethesoftwareto
7.1 Several different data sets are necessary to qualify an
AssistDR system for use in a production environment. These provide an incorrect result.
data sets correspond to the overall process described by Fig. 1 7.2.2 Astrongcross-functionalteamisnecessarytoperform
and are broken down into more detail in Fig. 7 below. In Fig. an effective FMEA. This team should include manufacturing
7,thefirststepistocarefullydefinethespaceofNDTdatathat process experts to identify type, size, and location of indica-
is representative of the manufacturing and inspection process. tions; inspection process experts to identify variation in image
E3327/E3327M−21
FIG. 7Data Sets Used in AssistDR System Qualification and Software Maintenance
view angle, quality, and noise; andAssistDR system experts to boundary (3, 4).There are areas of thinner material (2), thicker
identify weak points of algorithms and filters. Such a group is
material (3), and a transition from thin to thick (5).
needed to identify the broadest set of failure modes possible to
7.2.4 After the failure modes are identified, they should be
include in the training data set.
incorporated into the training and qualification databases for
7.2.3 The objective of a FMEA for the qualification and
the AssistDR system. For elements of the FMEA having a
control of AssistDR software is to identify the range of input
range of values, an efficient way to put together a data set that
data to the software that could impact performance. A more
spans the entire space is by applying the Design of Experi-
extensiveFMEAconsideringabroaderrangefactorsimpacting
ments (DOE) methodology. Design of Experiments is a sys-
implementation, such as computation time, network speeds,
tematic method to determine the relationship between factors
operator acceptance, etc., is recommended. The analysis in-
affecting a process and the output of that process. The most
cludes expected variations in the manufacturing process, in-
common DOE method uses a full factorial design where the
spection process, and software algorithms. The output of the
boundary points on the range of the input parameters are
FMEAshould be a table of factors that could cause a failure of
combined in a systematic manner to form a set of experiments.
the AssistDR software. An example table of the output of a
For the example in Table 3, this would result in a 48
FMEA is found in Table 3.
(2x2x2x2x3) run DOE for each indication type. Each of the
7.2.3.1 This example is based on the positive image in Fig.
runs for this design is shown in Table 4.
8 where air is shown as white and the part being inspected as
7.2.5 After creating a DOE design, images are assembled
shadesofgray.Thedarkershadesofgrayindicatethickerareas
representing each of the runs in the DOE. It is preferable for
ofthepart.Therearetwoindicationtypes(foreignmaterialand
these to be indications that are produced by the manufacturing
porosity) that theAssistDR system is required to detect. In this
process and created on the production inspection system. Due
example, each of these types has a 0.020 in. minimum
to the nature of manufacturing processes, not all indications
interpretable size, but the manufacturing process produces
that may occur in the process occur on a regular basis. These
indications over a wide range of sizes. Due to the nature of the
infrequently occurring indication types may make completing
foreign material and porosity, the digital signals produced by
the indications vary in both magnitude and signal-to- the DOE image matrix in a timely manner impossible. In these
background ratio. There are two boundaries of interest in the cases, it is necessary to use carefully created simulated images
image, the part/air boundary (1) and the part/end of image to fill out the DOE matrix.
E3327/E3327M−21
TABLE 3 Example FMEA Output For A Radiographic Inspection Process Based on Notional Image in Fig. 8
Impact of Failure
Failure Modes Range of Causing Condition Failure Mode Mitigation
Miss False Positive
Foreign Material Too Small to Detect × 0.020 in. Min Interpretable 10 indications between 0.025 in. and
0.015 in. major dimension in
qualification data set
Foreign Material Too Large to Detect × Indications greater than 0.150 in. 10 indications greater than 0.150 in.
may cause algorithm error in qualification data set
Low Contrast Foreign Material × Foreign material less than CNR 10 indications within 5 % of
threshold possible CNRthreshold in qualification data
set
High Contrast Foreign Material × Large areas of foreign material rare 10 indications greater than 200 % of
but possible CNRthreshold in qualification data
set
Porosity Too Small to Detect × 0.020 in. Min Interpretable 10 indications between 0.025 in. and
0.015 in. major dimension in
qualification data set
Porosity Too Large to Detect × Indications greater than 0.150 in. 10 indications greater than 0.150 in.
may cause algorithm error in qualification data set
Low Contrast Porosity Detection × Porosity less than CNR threshold 10 indications between within 5 % of
rare but possible CNRthreshold in qualification data
set
High Contrast Porosity Detection × Large pores rare but possible 10 indications greater than 200 % of
CNRthreshold in qualification data
set
Algorithm Performance at Part/Air Interfaces × × 2 regions of part/air interface 20 indications located within 0.010 in.
of a part/air interface in qualification
data set
Algorithm Performance at Part/Image Edge × × 2 regions of part/image edge 20 indications located within 0.010 in.
Interfaces interface of a part/image edge interface in
qualification data set
Algorithm Performance in Areas of Thickness × × 1 region of thickness transition 20 indications located within 0.020 in.
Transition between platform and airfoil of a thickness transition in
qualification data set (10 in thicker
region, 10 in thinner in region)
Complete Detector Failure × All data in image occupies less than Error check for detector failure
5 % of dynamic range incorporated into software before
detection algorithm
Detector Degradation × × Areas of low contrast or higher than Performance check of system using
acceptable noise a reference part at the start and end
of each shift
Operator Does Not Evaluate Identified Indication × × All software identified indications Software implements check box for
must be evaluated each indication and all check boxes
must be set before software will
register the inspection complete
7.2.6 TheresultsoftheAssistDRsystemontheDOEshould 7.2.8 The nature of the assessment data set used in a
be expressed as a TPR for the indications. While other metrics qualification study is vital to determining the efficacy of an
such as FPR may be of interest and provide insight into the AssistDR system. This data set should consist of a sufficiently
system’s performance, the DOE design method described in large (see discussion in 6.4) collection of indications and
this section is intended to investigate the impact of process inspection opportunities to provide estimates of statistical
variation on TPR. sensitivity and specificity for a purely manual inspection
7.2.7 Requirements on the performance of the AssistDR system and the associated AssistDR-enabled inspection sys-
system on the DOE matrix will vary from project to project. If tem. In practice, determining how many indications are re-
a large, statistically significant DOE matrix like the one quired is often a tradeoff between statistical requirements and
specified in Table 4 is used, a TPR requirement based on the practical matters such as availability of indications in
production inspection processes can be applied to evaluate the production, operator availability, cost, etc. If a reasonable
DOE results. For some projects, a statistically significant DOE initial estimate of operator sensitivity and specificity is
may not be possible. In this case, the experiment can use a available, the method described in 6.4.2 can be employed to
reduced size DOE focused on specific indication conditions identify the number of indications and indication-free inspec-
(for example, location, contrast, size) that do not occur in tion system outputs required to provide real sensitivity and
typical production but are identified by the FMEA as condi- specificity estimates with a specified degree of uncertainty.
tions the algorithm needs to address. If a reduced size DOE is Generally speaking, if a statistician is unavailable, it is prudent
used, a higher TPR should be required of the software on the to err on the side of larger data sets. Beyond the quantity of
DOE data to ensure that theAssistDR system has some degree indications and indication-free inspection opportunities, it is
of sensitivity to those conditions. extremely important to ensure that the assessment data set
E3327/E3327M−21
FIG. 8Image for FMEA Example
TABLE 4 Example DOE based on FMEA in Table 3
Run Size SBR Image Part Gradient Run Size SBR Image Location Part Gradient
Order Location Location Order Location
1 Small Low Edge Edge B->G 25 Large Low Edge Edge B->G
2 Small Low Edge Edge W->G 26 Large Low Edge Edge W->G
3 Small Low Edge Edge Flat 27 Large Low Edge Edge Flat
4 Small Low Edge Center B->G 28 Large Low Edge Center B->G
5 Small Low Edge Center W->G 29 Large Low Edge Center W->G
6 Small Low Edge Center Flat 30 Large Low Edge Center Flat
7 Small Low Center Edge B->G 31 Large Low Center Edge B->G
8 Small Low Center Edge W->G 32 Large Low Center Edge W->G
9 Small Low Center Edge Flat 33 Large Low Center Edge Flat
10 Small Low Center Center B->G 34 Large Low Center Center B->G
11 Small Low Center Center W->G 35 Large Low Center Center W->G
12 Small Low Center Center Flat 36 Large Low Center Center Flat
13 Small High Edge Edge B->G 37 Large High Edge Edge B->G
14 Small High Edge Edge W->G 38 Large High Edge Edge W->G
15 Small High Edge Edge Flat 39 Large High Edge Edge Flat
16 Small High Edge Center B->G 40 Large High Edge Center B->G
17 Small High Edge Center W->G 41 Large High Edge Center W->G
18 Small High Edge Center Flat 42 Large High Edge Center Flat
19 Small High Center Edge B->G 43 Large High Center Edge B->G
20 Small High Center Edge W->G 44 Large High Center Edge W->G
21 Small High Center Edge Flat 45 Large High Center Edge Flat
22 Small High Center Center B->G 46 Large High Center Center B->G
23 Small High Center Center W->G 47 Large High Center Center W->G
24 Small High Center Center Flat 48 Large High Center Center Flat
covers the full space of possible inspection contexts. Thus, qualificationdataset.Assemblingthisgroundtruthischalleng-
when developing an assessment data set, it is recommended ing because it requires that all indications in the image data set
that the AssistDR system purchaser or provider conduct an
larger than some minimum interpretable size be located,
FMEA Design of Experiments as described above.
measured, and assigned a unique identification. Doing this task
accurately usually requires significant manual intervention.
7.3 Ground Truth—Assembly of the associated ground truth
The accuracy of the performance measurements for the As-
data set is one of the most important, most time consuming,
sistDRalgorit
...