EN ISO 16140:2003

SLOVENSKI STANDARD
01-november-2003
Mikrobiologija živil in krme - Protokol za validacijo alternativnih metod (ISO
16140:2003)
Microbiology of food and animal feeding stuffs - Protocol for the validation of alternative
methods (ISO 16140:2003)
Mikrobiologie von Lebensmitteln und Futtermitteln - Arbeitsvorschrift für die Validierung
alternativer Verfahren (ISO 16140:2003)
Microbiologie des aliments - Protocole pour la validation des méthodes alternatives (ISO
16140:2003)
Ta slovenski standard je istoveten z: EN ISO 16140:2003
ICS:
07.100.30 Mikrobiologija živil Food microbiology
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

EUROPEAN STANDARD
EN ISO 16140
NORME EUROPÉENNE
EUROPÄISCHE NORM
May 2003
ICS 07.100.30
English version
Microbiology of food and animal feeding stuffs - Protocol for the
validation of alternative methods (ISO 16140:2003)
Microbiologie des aliments - Protocole pour la validation Mikrobiologie von Lebensmitteln und Futtermitteln -
des méthodes alternatives (ISO 16140:2003) Arbeitsvorschrift für die Validierung alternativer Verfahren
(ISO 16140:2003)
This European Standard was approved by CEN on 6 July 2002.
CEN members are bound to comply with the CEN/CENELEC Internal Regulations which stipulate the conditions for giving this European
Standard the status of a national standard without any alteration. Up-to-date lists and bibliographical references concerning such national
standards may be obtained on application to the Management Centre or to any CEN member.
This European Standard exists in three official versions (English, French, German). A version in any other language made by translation
under the responsibility of a CEN member into its own language and notified to the Management Centre has the same status as the official
versions.
CEN members are the national standards bodies of Austria, Belgium, Czech Republic, Denmark, Finland, France, Germany, Greece,
Hungary, Iceland, Ireland, Italy, Luxembourg, Malta, Netherlands, Norway, Portugal, Slovakia, Spain, Sweden, Switzerland and United
Kingdom.
EUROPEAN COMMITTEE FOR STANDARDIZATION
COMITÉ EUROPÉEN DE NORMALISATION
EUROPÄISCHES KOMITEE FÜR NORMUNG
Management Centre: rue de Stassart, 36  B-1050 Brussels
© 2003 CEN All rights of exploitation in any form and by any means reserved Ref. No. EN ISO 16140:2003 E
worldwide for CEN national Members.

Contents
page
Foreword. 3
Introduction . 4
1 Scope. 5
2 Normative references . 5
3 Terms and definitions. 6
4 General principles for the validation and the certification of alternative methods. 7
5 Qualitative methods - Technical protocol for their validation.8
6 Quantitative methods -Technical protocol for their validation . 18
Annex A (normative)  Specific rules for the acceptance of external results already obtained in a prior
validation scheme . 33
Annex B (informative)  Classification of sample types for validation studies . 35
Annex C (normative)  Use of naturally contaminated samples and preparation of artificially
contaminated samples in validation studies. 38
Annex D (normative)  Duplication of samples for the determination of relative accuracy and of relative
detection level for qualitative methods . 40
Annex E (normative)  Calculation of the confidence intervals associated with the number of samples
tested. 42
Annex F (normative)  Test applied to the examination of discordant results. 43
Annex G (normative)  Points to be considered when selecting strains for testing selectivity . 44
Annex H (normative)  Guidelines for the organisation and conducting collaborative studies. 46
Annex I (normative)  Determination that negative controls are free of target analyte. 49
Annex J (normative)  Replication of samples for interlaboratory studies of qualitative methods . 50
Annex K (normative)  Consideration of data .52
Annex L (informative)  Interlaboratory study of qualitative methods: criteria of accordance,
concordance and concordance odds ratio . 53
Annex M (normative)  Replication of samples for the determination of relative accuracy of quantitative
methods . 58
Annex N (normative)  Examples of acceptable and unacceptable situations and range of
measurements for the estimation of the regression line for quantitative methods. 60
Annex O (normative)  Assessment of the linearity of quantitative methods by graphical representation. 62
Annex P (normative)  Detection and quantification limits for counts. 63
Annex Q (normative) Robust estimator of dispersion based on the recursive median Sn from
Rousseeuw [6]. 65
Annex R (normative)  Calculations with the regression method. 66
Annex S (normative)  Examples of calculations for quantitative methods . 71
Annex T (normative)  Collaborative study – Ring test results with duplicates. 76
Annex U (informative)  List of symbols and abbreviations . 77
Bibliography . 78
Foreword
This document (EN ISO 16140:2003) has been prepared by Technical Committee CEN/TC 275 "Food analysis -
Horizontal methods", the secretariat of which is held by DIN, in collaboration with Technical Committee ISO/TC 34
"Agricultural food products".
This European Standard shall be given the status of a national standard, either by publication of an identical text or
by endorsement, at the latest by November 2003, and conflicting national standards shall be withdrawn at the latest
by November 2003.
The annexes A, C to K and M to T are normative. The annexes B, L and U are informative.
This document contains also a Bibliography.
According to the CEN/CENELEC Internal Regulations, the national standards organizations of the following coun-
tries are bound to implement this European Standard: Austria, Belgium, Czech Republic, Denmark, Finland,
France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Luxembourg, Malta, Netherlands, Norway, Portugal,
Slovakia, Spain, Sweden, Switzerland and the United Kingdom.
Introduction
The need for the food industry to rapidly assess the microbiological quality of raw materials and finished products
and the microbiological status of manufacturing procedures, has led to the development and refinement of alterna-
tive microbiological methods of analysis that are quicker and/or easier to perform than the corresponding reference
method; some can also be automated.
Among these alternative methods, some can yield results that are equivalent to those provided by the reference
method, while others can lead to results that differ appreciably.
The suppliers/producers of the alternative methods, the food and drink industry, the public health services and
other authorities need a reliable common protocol for the validation of such alternative methods. The data gener-
ated can also be the basis for the certification of a method by an independent organisation.
Because of the extent of the methods comparative study described in this standard for use by the organising labo-
ratory, the procedure is sometimes not appropriate for use as an "in house" method for the validation of an alterna-
tive method by an individual laboratory.
1 Scope
This document establishes the general principle and the technical protocol for the validation of alter-
native methods in the field of microbiological analysis of food, animal feeding stuff and environmental
and veterinary samples (see 5.1.1.2.1) for:
 the validation of alternative methods which can be used in particular in the framework of the offi-
cial control;
 the international acceptance of the results obtained by the alternative method.
It also establishes the general principles of certification of these alternative methods, based on the
validation protocol defined in this EN ISO 16140 (see 4.2).
Where an alternative method is used on a routine basis for internal laboratory use without the re-
quirement to meet (higher) external criteria of quality assurance, a less stringent comparative valida-
tion of the alternative method than that set in this standard may be appropriate.
2 Normative references
This European Standard incorporates by dated or undated reference, provisions from other publica-
tions. These normative references are cited at the appropriate places in the text, and the publications
are listed hereafter. For dated references, subsequent amendments to or revisions of any of these
publications apply to this European Standard only when incorporated in it by amendment or revision.
For undated references the latest edition of the publication referred to applies (including amend-
ments).
ISO 3534-1, Statistics – Vocabulary and symbols – Part 1: Probability and general statistical terms.
ISO 5725, Accuracy (trueness and precision) of measurement methods and results.
3 Terms and definitions
For the purposes of this European Standard, the following terms and definitions apply:
3.1
alternative method
method of analysis that demonstrates or estimates, for a given category of products, the same analyte
(3.4) as is measured using the corresponding reference method (3.2).
NOTE 1 The method can be proprietary or non commercial, and does not need to cover an entire analysis pro-
cedure, that is from the preparation of samples to the test report.
NOTE 2 The alternative method exhibits attributes appropriate to the users' needs, for example:
 speed of analysis and/or response;
 ease of execution and/or automation;
 analytical properties (precision, accuracy, limit of detection, etc.);
 miniaturisation;
 reduction of cost.
NOTE 3 The term "alternative" is used to refer to the entire "test procedure and reaction system". This term
includes all ingredients whether material or otherwise, required for implementing the method.
3.2
reference method
internationally recognised method and widely accepted.
NOTE For the purpose of this standard, these are International and European Standards and if not existing,
certain national standards of equivalent standing.
3.3
validation of an alternative method
demonstration that adequate confidence is provided that the results obtained by the alternative
method are comparable to those obtained using the reference method
NOTE The word "comparable" is defined in this EN ISO 16140 by a technical protocol adapted to each type
of method (see clauses 5 and 6).
3.4
analyte
component measured by the method of analysis. It may be the microorganism
3.5
qualitative method
method of analysis whose response is either the presence or absence of the analyte (3.4) detected
either directly or indirectly in a certain amount of sample
3.6
quantitative method
method of analysis whose response is the amount of the analyte (3.4) measured either directly (enu-
meration in a mass or a volume), or indirectly (colour absorbance, impedance, etc.) in a certain
amount of sample
3.7
methods comparison study
study, performed by the organising laboratory of the alternative method against the reference method
3.8
inter-laboratory study
study of the method’s performance using common samples in several laboratories and under the con-
trol of the organising laboratory
3.9
organising laboratory
laboratory having the qualified staff and skills to perform the method comparison study and organise
the interlaboratory study.
NOTE The availability of an experienced statistician is essential for the analysis of the results.
4 General principles for the validation and the certification of alternative
methods
4.1 Validation protocol
The validation protocol comprises two phases:
 a methods comparison study (3.7) of the alternative method (3.1) against the reference method
(3.2) carried out in the organizing laboratory;
 an interlaboratory study (3.8) of each of the two methods.
If appropriate, the two phases may be undertaken in parallel.
The technical rules for performing the methods comparison study and the interlaboratory study are
given in clauses 5 and 6, depending upon whether the alternative method is qualitative or quantitative
in nature.
If the alternative method has already been validated and meets the requirements set by another or-
ganisation, specific rules are defined in annex A for accepting the results of this prior validation.
4.2 Principles of the certification
4.2.1 If a subsequent certification of the alternative method is required, the two following principles
shall also be applied (in addition to 4.1):
Details on the organisation of the certification (management of the method comparison study and the
interlaboratory study, all the different bodies involved including the expert laboratory – designated in
this standard as the "organising laboratory"- the reviewers, the certification body, etc) are provided [8]
by the certification body.
4.2.2 The manufacturer shall apply a quality system covering the production line of the product for
which the certification is sought and based on the appropriate European Standard relative to quality
systems or other equivalent international standard (for example EN ISO 9001).
In granting the certification, the certification organisation shall take into account the existence of any
quality system certificate issued by a certification body accredited for quality systems.
4.2.3 A regular verification of the quality of the certified method shall be undertaken after the certi-
fication is granted. An audit is to be performed regularly to verify that the following are still met:
 the quality assurance requirements, (see 4.2.1);
 the product's production control requirements, (see 4.2.1).
In addition to the general requirements of the appropriate European Standard relative to the quality
system, the manufacturer presents regularly to the certification organisation updated documentation
that take into account any modification made to the product or production process which may affect
the instructions for using the method and/or the method’s performance. The certification organisation
then decides whether these modifications affect the certification.
5 Qualitative methods - Technical protocol for their validation
5.1 Methods comparison study
5.1.1 Relative accuracy, relative specificity and relative sensitivity
5.1.1.1 Terms and definitions
For the purposes of this European Standard, the following terms and definitions apply.
AC
5.1.1.1.1 relative accuracy ( )
degree of correspondence between the response obtained by the reference method and the response
1)
obtained by the alternative method on identical samples (see 5.1.1.3.1.).
NOTE The term "relative accuracy" used here is complementary to the "accuracy" and “trueness” as defined in
ISO 5725-1 and ISO 3534-1. These state that accuracy is "the closeness of agreement between a test result and
the accepted reference value", and that the trueness is "the closeness of agreement between the average value
obtained from a large series of test results and an accepted reference value". For the purpose of this standard,
the accepted reference value is chosen as the value obtained by the reference method. Thus, the term "relative"
implies that the reference method does not automatically provide the accepted reference value.
5.1.1.1.2 positive deviation (PD)
The alternative method becomes a false positive when it presents a positive deviation if it gives a
positive result when the reference method gives a negative result.
A positive deviation becomes a false positive result when the true result can be proven as being nega-
tive.
A positive deviation is considered as a true positive when the true result can be proven as being posi-
tive.
5.1.1.1.3 negative deviation (ND)
The alternative method presents a negative deviation if it gives a negative result when the reference
method gives a positive result.
A negative deviation becomes a false negative result when the true result can be proved as being
positive.
1)
Difficult to achieve if the pre-enrichment steps are different.
5.1.1.1.4 relative sensitivity (SE)
ability of the alternative method to detect the analyte when it is detected by the reference method
(see 5.1.1.3.1.).
5.1.1.1.5 relative specificity (SP)
ability of the alternative method to not detect the analyte when it is not detected by the reference
method (see 5.1.1.3.1.).
5.1.1.2 Measurement protocol
5.1.1.2.1 Food samples
It is of the highest priority to find food samples naturally contaminated with the analyte to be detected
for the validation.
If it is sought to validate the method for all foods, study five categories of food. This number may be
reduced to 1, 2, 3 or 4 categories if the validation of alternative method is restricted to these stated
categories, at the producer's request. The recommended categories are listed in annex B.
Appropriate environmental samples may be included as one category. Veterinary samples may be
treated as another category (see annex B).
It is desirable that food samples come from as wide a distribution as possible in order to reduce any
bias from local food specialities and broaden the range of validation.
When analysing naturally contaminated samples, the range and distribution of contamination of the
samples should be representative of the levels usually found in that product but with emphasis on
smaller numbers.
If it is not possible to acquire a sufficient number of naturally contaminated foods for each of the cate-
gories, artificial contamination of food samples is permissible. The method and levels of contamination
should result in samples behaving similarly to naturally contaminated ones. See methods of inocula-
tion and restrictions in annex C.
5.1.1.2.2 Number of samples
The total number of test portions to be analysed is 60 for each food category chosen from the catego-
ries stated in annex B. Within each category, select representative food types and analyse 20 test
portions of each food type by the proposed method and the reference method to produce at least
60 total results for each category by each method. For naturally contaminated food types prepare the
sample as described in annex D. For artificially contaminated food type adjust the inoculation levels to
achieve fractional positive recovery of the test portions analysed by at least one of the methods. Frac-
tional recovery is achieved when some number, but not all, of the test portions are determined to be
positive by one or both methods, alternative method or reference method.
It is desirable to produce approximately 50 % of the results that are positive and 50 % that are nega-
tive. This is, however, a recommendation, not an absolute percentage, provided that some number of
the test portions are positive and some number are negative for the same food type.
5.1.1.2.3 Test sample preparation
The reference and alternative methods shall be performed with, as far as possible, exactly the same
sample.
Thus, if the first stage of the two methods is the same (for example the same pre-enrichment broth),
perform the replication at the second step (case 1, annex D).
If this is not the situation, that is the first culture media, methodology or dilutions are different, prepare
paired test portions for analysis. There are two primary methodologies for such preparations.
In the first instance, mix a double weight of sample with an equal weight/volume of sterile water or
other suitable diluents and homogenize very thoroughly. Then divide into two portions taking particular
care to increase the concentration of the primary enrichment by (approximately 10 %) to compensate
for the dilution effect of the diluted, homogenised sample (case 2, annex D).
In the second instance, directly inoculate the food type with a starting inoculum sufficient to allow a
fractional recovery of the micro-organisms in the test portions analysed by at least one of the methods
after the microorganisms have equilibrated in the food type. Then weigh 25 g test portions and pro-
ceed as described in annex D. This may be preferred for liquid products but is acceptable for any food
type provided that the food is properly homogenised.
5.1.1.3 Calculation and interpretation
5.1.1.3.1 Treatment of data
Tabulate the data of the paired results of the reference and alternative methods and calculate the fol-
lowing parameters for each food category (60 samples) according to the Table 1.
Table 1 - Paired results of the reference and alternative method
Responses Reference method positive (R+) Reference method negative (R-)
Alternative method posi- +/+ positive agreement (PA) -/+ positive deviation (PD) (R-/A+)
tive (A+)
Alternative method nega- +/- negative deviation (ND) (A-/R+) -/- negative agreement (NA)
tive (A-)
The calculations shall be performed on a number of negative results obtained by the reference method
which for the results in Table 1 cannot exceed twice the number of positive results; the negative re-
sults being selected if necessary as immediately following a positive result, in the order of analysis of
the samples.
Express the three criteria as follows:
()PA + NA
 AC = ·100 % ;
Relative accuracy:
N
NA
 SP = ·100 % ;
Relative specificity:
N
-
PA
 Relative sensitivity: SE = ·100%
N
+
where
N is the total number of samples (NA + PA + PD +ND);
N is the total number of negative results with the reference method (NA + PD);
-
N is the total number of positive results with the reference method (PA + ND).
+
5.1.1.3.2 Confidence intervals
The calculation of confidence intervals associated with the number of samples tested is given in an-
nex E.
5.1.1.3.3 Discordant results
Examine the discordant results as described in annex F (The McNemar test), by using the count of PD
and ND (see 5.1.1.3.1).
When the values for PD and ND are high and almost equal, no statistical difference between the
methods can be detected using the McNemar test. In this case, the organising laboratory shall pay
further attention to explain the reasons for the high values of PD and ND. Moreover, it shows that the
relative accuracy of a method shall never be interpreted by taking into account only the McNemar test.
5.1.1.3.4 Summary of calculation
All the calculations shall be summarised in Table 2:
Table 2 - Calculation of the relative accuracy, the relative sensitivity and the relative specificity
Matrices PA NA ND PD Sum Relative Accuracy N Relative sensitivity N Relative
+ -
AC (%) SE (%) specificity SP
(%)
100·()PA + NA 100·PA 100·NA
N PA + ND NA + PD
N N+ N-
Food cat. 1
Food cat. 2
Food cat. 3
Food cat. 4
Food cat. 5
TOTAL
5.1.1.3.5 Interpretation
A table giving the raw results (that is all the positive and negative results, Table 1) shall be provided.
Taking into account the number of positive deviations and the number of negative deviations, the ca-
pability of the alternative method to give more or fewer true positive results than the reference method
is evaluated.
The report of the study shall distinguish the results obtained with naturally contaminated and artificially
contaminated samples.
The procedure for the artificial contamination of test samples shall be described in the report of the
study.
Data published elsewhere and meeting the conditions defined in annex A may be used for evaluating
the relative accuracy.
5.1.2 Relative detection level
5.1.2.1 Definition
For the purpose of this standard, the relative detection level is the smallest number of culturable mi-
croorganisms (3.4) that can be detected in the sample in 50 % of occasions by the alternative and ref-
erence methods.
5.1.2.2 Measurement protocol
Test the following:
 use one food product within each food category chosen from 5.1.1.2.1, depending of the scope of
the validation (see annex B);
 use five different target microorganisms (or less, depending on the scope of the validation) each
one associated with one food category, if possible. (See annex G.1 for the definition of the target
microorganism);
 preferably test five levels (but a minimum of three levels) of one target microorganism per food,
including the negative control, etc. The first level shall be the negative control. The second level
shall be the theoretical detection level. The third level shall be just above the theoretical detection
threshold and any further levels shall be higher than the previous one. A factor of about three
between each concentration in the upper levels could be applied;
 replicate each combination (food product, level of contamination) six times by both the alternative
and reference methods. Perform the division at the level where the two methods differ as illus-
st
trated in annex D. Thus, if the 1 stage of each method is the same (for example the same pre-
nd
enrichment broth), perform the division at the 2 step (case 1, annex D). If this is not the case, i.e.
the first culture media, methodology or dilutions being different, mix a double weight of sample
with an equal w/v of sterile water or other suitable diluent and then divide into two portions;
 apply the complete procedure of the alternative method and the reference method, including the
preparation of the sample. Inoculation of each food sample may be prior to its addition to the cul-
ture medium or afterwards.
If necessary, for assuring a better precision of the lowest inoculum level, increase the amount of food
sample or the number of replicate samples. For example, 75 g of food sample contaminated with three
cells instead of 25 g contaminated with one cell.
The greater the number of inoculum levels used the more precise is the determination of the detection
threshold.
5.1.2.3 Calculation
For each level L (i = 0 to 3) and each food/strain combination (j = 1 to 5), compare both methods as
i
stated in Table 3:
Table 3 - Calculation of relative detection level
Results
Negatives (-) Positives (+) Total
Method
Reference an - an=6
Alternative b n – bn=6
a+b 2n - (a + b)2n=12
Total
For small 2 by 2 tables, perform exact Fisher tests [8].
Comparisons
Instead of only comparing both methods at each level and each food/strain, the same test to compare
two food/strains at the same level can be used.
If food/strains seem to be comparable, the same test is available with n > 6 in pooling food/strains for
each level L .
i
The levels can also be pooled to do checks, but using the ranking order: L + L , L + L + L , L + L , L
0 1 0 1 2 1 2 0
+ L + L + L , L + L + L , L + L … with or without pooling the food/strains.
1 2 3 1 2 3 2 3
Report all the significant differences between methods, food/strains and/or levels.
5.1.2.4 Interpretation
The interpretation shall be done by the organising laboratory in charge of the methods comparison
study.
The relative detection level lies between the two contamination levels giving respectively less and
more than 50 % detection level. The relative detection level is therefore expressed as a range.
5.1.3 Inclusivity and exclusivity
5.1.3.1 Definition
Inclusivity is the ability of an alternative method to detect the target analyte from a wide range of
strains.
Exclusivity is the lack of interference from a relevant range of non-target strains of the alternative
method.
5.1.3.2 Measurement protocol
5.1.3.2.1 Selection of test strains
5.1.3.2.1.1 General
For microorganisms a range of strains is chosen to avoid any local bias.
Criteria for selecting test strains are given in annex G.
Each strain shall be characterised biochemically, serologically and if relevant genetically, in sufficient
detail for its identity to be established and should be preferentially isolated from food. Also the food
material from which it was originally isolated shall be known and recorded.
5.1.3.2.1.2 Target microorganisms
Select at least 50 pure cultures of microorganisms relevant to the alternative method and the food
product being used (see G.3), except for Salmonella.
For Salmonella methods, select at least 30 pure cultures of microorganisms.
5.1.3.2.1.3 Non-target microorganisms
Select at least 30 pure cultures of microorganisms chosen from both the strains known to cause in-
terference with the target microorganism and from strains naturally present in each food test material
included in the validation (see G.4).
5.1.3.2.2 Inoculation
5.1.3.2.2.1 General
Each test is performed once. Inoculation of the growth medium is carried out using a dilution of a pure
culture of each test strain. No food sample is added.
5.1.3.2.2.2 Target microorganisms
The inoculum level shall be 10 times to 100 times greater than the minimum relative detection level of
the alternative method and the complete protocol of the alternative method shall be used, including
pre-enrichment if stipulated. When false negative or doubtful results are obtained, the strain shall be
tested once more together with the reference method.
5.1.3.2.2.3 Non-target microorganisms
The inoculum level of a strain shall be similar to the greatest level of contamination expected to occur
in all the food categories being used.
The exclusivity shall be established. If the final nutrient medium of the culture medium is a selective
broth this would be replaced by an appropriate non selective broth medium. When the alternative
method gives positive or doubtful results with non-target microorganisms, the test shall be repeated
using the complete protocol. The reference method shall be performed only once.
5.1.3.3 Expression of the results
Tabulate the results as in Table 4:
Table 4 - Presentation of the results for the selectivity
Microorganisms Results
Reference method Alternative method
Expected result Actual result Expected result Actual result
Target strains
etc.
Non target strains
etc.
5.1.3.4 Interpretation
The interpretation shall be done by the laboratory in charge of the methods comparison study, as both
quantitative and qualitative aspects (that is pathogenicity, prevalence, cultural aspects of test strains,
for example motility, sensitivity to inimical agents etc.) shall be taken into account.
Other published data on the alternative method that meets the requirements of this EN ISO 16140,
may also be used by the laboratory in charge of the methods comparison study, to provide further in-
formation on the above criteria. (See annex A that provides criteria for the acceptance of external re-
sults).
5.2 Interlaboratory study
NOTE The aim of the collaborative study is to determine the variability of the results obtained in different
laboratories using identical samples and to compare these results with those obtained in the methods comparison
study.
5.2.1 Measurement protocol
5.2.1.1 The interlaboratory study shall produce at least 10 collaborative laboratories having re-
sults without outliers.
Guidelines and requirements for the organisation, dispatching and conducting the interlaboratory
studies are given in annex H.
It is necessary for the analyst in each collaborating laboratory to demonstrate his competence in the
use of the alternative method and of the reference method prior to participating in the study proper.
5.2.1.2 The protocol is the following:
 one relevant food matrix (annex B) is used to prepare the test samples;
 the protocol for artificial contamination of the food sample shall be appropriate for the selected
food substrate. Each sample shall be individually inoculated. Each blind replicate shall be pre-
pared to ensure homogeneity between samples by the individual inoculation of each one. An ap-
propriate number of samples shall be analysed to determine homogeneity (see annex H);
 at least three different levels of contamination shall be used: a negative control (L ), one level
slightly above the detection level of the alternative method (L ), and one about 10 times greater
than the detection level (L ) (for example 0; 3; and 30 cells/25g);
 at least 8 blind replicates at each level of contamination are analysed by both reference and alter-
native methods by each collaborative laboratory;
 a suspension of the whole of each sample is prepared for analysis;
 the analysis of samples shall be performed in each laboratory at the stipulated date;
st
 the test samples are cultured according to annex J. Thus if the 1 culture step is common to both
reference and alternative methods use this culture to inoculate each of the next stages (case 1,
annex J). If the primary cultures of each method differ, then replicate by setting up each method
individually (case 2, annex J);
 in either case, the combination "number of levels of contamination/number of replicates/number of
non-outlier laboratories" shall be selected so that at least 480 results (240 by each method) are
generated for use in the calculations.
The organising laboratory using all recorded data (see H.3) shall determine which results
5.2.1.3
are suitable and which are outliers for use in calculating the precision data. See annex K, that provides
guidelines defining microbiological conditions for disregarding data.
WARNING Outliers: Do not exclude participating laboratory results if there is no clear expla-
nation or gross error to explain them.
5.2.2 Calculation
5.2.2.1 For each level, put the positive results obtained with each method as in Tables 5 and 6:
Table 5 – Positive results by the reference method
Contamination level
Laboratories L L L
0 1 2
Laboratory 1 /8 /8 /8
Laboratory 2 /8 /8 /8
Laboratory 3 /8 /8 /8
etc. /8 /8 /8
a b c
Total FP TP TP
1 2
a
False positive by the reference method
b
True positive at level 1 by the reference method
c
True positive at level 2 by the reference method
Table 6 – Positive results by the alternative method
Contamination level
Laboratories L L L
0 1 2
Laboratory 1 /8 /8 /8
Laboratory 2 /8 /8 /8
Laboratory 3 /8 /8 /8
etc. /8 /8 /8
a b c
Total FP TP1 TP2
a
False positive by the alternative method
b
True positive at level 1 by the alternative method
c
True positive at level 2 by the alternative method
5.2.2.2 For level L and for each method, calculate the percentage specificity SP as in equation
(1):
 FP 
 
SP = 1-·100 % (1)
 
 
N-ŁłŁł
where
N is the total number of all L tests;
- 0
FP is the number of false positive.
5.2.2.3 For each contamination level and for each method, calculate the percentage of sensitivity
SE as in equation (2):
TP
SE =·100 % (2)
N
+
where
N is the total number of all L or L tests respectively;
+ 1 2
TP is the number of true positive.
5.2.2.4 For each level of contamination and the totality of the results, compare the alternative
method and the reference method in order to calculate the relative accuracy and to examine the dis-
cordant results.
Each pair of results from a sample measured by the alternative and the reference method shall be re-
ported as in Table 7:
Table 7 - Paired results of the alternative method and the reference method by the interlabora-
tory study
Alternative Reference method Total
method
+-
+PA PD
-ND NA
Total N N N
+ -
Calculate the relative accuracy AC expressed in percentage, as in equation (3):
(PA + NA)
AC =·100 % (3)
N
where
N is the number of tested samples (for the level L or all levels);
i
PA is the number of positive agreement;
NA is the number of negative agreement.
5.2.2.5 Calculate the confidence intervals for each proportion (see 5.1.1.3.2).
5.2.2.6 Examine the discordant results as described in annex F, by using the counts of PD and
ND (see Table 8).
5.2.3 Interpretation
Compare AC (Table 7), SE and SP (Tables 5 and 6) with their relative counterparts obtained within the
comparative study including the naturally contaminated samples and the relative detection level.
These criteria do not really address the variability within a laboratory and between laboratories of the
method (notions of repeatability and reproducibility). Annex L provides further criteria (accordance,
concordance and concordance odds ratio) which may help to address this variability (the criteria of
repeatability and reproducibility have been defined for quantitative methods and cannot be used as
such for qualitative methods).
6 Quantitative methods - Technical protocol for their validation
6.1 General
Colony counts from a given sample are the most common output from reference quantitative methods
and all information should be recorded – sample weight, dilution series, inoculum volume and colony
counts at each dilution. In microbiology, these discrete data are often truncated, for example by only
counting plates with no more than 300 colonies and then applying an expansion factor inverse to the
dilution factor to obtain the result. For this reason and other factors involving the growing process of
the microorganisms, and also in order to obtain a symmetric distribution or a near normal one, the
counting data are often transformed into their logarithms, or a square-root transformation. This trans-
formation can be checked by using histograms of many data points (30 or more) all gathered under
the same conditions.
In 6.2 and 6.3 for the validation of quantitative methods the gathering of continuous data (or interval
data) is mainly described; however remarks about counts are included as a particular category.
6.2 Methods comparison study
6.2.1 Linearity and relative accuracy
6.2.1.1 Definitions
6.2.1.1.1 Linearity
Ability of the method when used with a given matrix to give results that are in proportion to the amount
of analyte present in the sample, that is an increase in analyte corresponds to a linear or proportional
increase in results.
NOTE 1 A response curve or a signal function is obtained when measuring the relationship between the
signal or the method’s response and the analyte concentration (doses) in different samples of reference ma-
terials (RM) having known values. In microbiology, where practically no stable reference material is available,
these "known values” can be obtained after many replicated measurements using the reference method.
After data fitting, smoothing or another algorithm, the alternative method provider should establish a monotonic
model over the whole application domain of the method in order to transform the measurements into the nearest
values to the reference ones. The fitted model is a first (or original) calibration curve and in collecting all the
calibration factors over the concentration domain; it is often not linear. But this is not included in the validation
study.
A part of the validation is the verification of the calibration, which gives a final calibration curve setting the
2)
relationship between the transformed measurements and the corresponding "reference values" , with the same
unit system. With the same scales on the axes, this curve should be linear, having a null intercept (same lowest
values), and a slope of 1 (same axes units) and with well estimated characteristics of spread. Extrapolation above
and below the tested concentrations should not be performed, except if it is necessary to examine the behaviour
near the concentration "zero”.
NOTE 2 Counts by the following regression method the linearity is not correctly obtained for counts with low
levels or with wide ranges. These counts have quasi-poissonian distributions with repeatability standard devia-
tions proportional to the square root of the mean counts. It involves the same kind of estimation difficulty as men-
tioned under note in 6.2.3.1.
2)
They are derived from the reference method with naturally contaminated samples, if RM's and «known» val-
ues are not available.
6.2.1.1.2 Accuracy
closeness of agreement between a test result and the accepted reference value [ISO 3534-1]
NOTE The term accuracy, when applied to a set of test results, involves a combination of random compo-
nents and a common systematic error or bias component.
6.2.1.1.3 Bias
difference between the expectation of the test results and an accepted reference value [ISO 3534-1]
NOTE Bias is the total systematic error as contrasted to random error. There can be one or more systematic
error components contributing to the bias. A larger systematic difference from the accepted reference value is
reflected by a larger bias value.
6.2.1.1.4 Relative accuracy
See 5.1.1.1.1
6.2.1.2 Measurement protocol
6.2.1.2.1 Design
The verification of the calibration curve requires many different samples (if possible reference materi-
als; see annex C); and the number of levels examined (doses or concentrations) is a function of the
range of concentrations to be used in practice.
A minimum of fi
...