EN ISO 16140-5:2020

SLOVENSKI STANDARD
01-november-2020
Mikrobiologija v prehranski verigi - Validacija metode - 5. del: Protokol za
medlaboratorijsko validacijo nelastniških metod (ISO 16140-5:2020)
Microbiology of the food chain - Method validation - Part 5: Protocol for factorial
interlaboratory validation for non-proprietary methods (ISO 16140-5:2020)
Mikrobiologie der Lebensmittelkette - Verfahrensvalidierung - Teil 5: Arbeitsvorschrift für
eine faktorielle Ringversuch-Verfahrensvalidierung (ISO 16140-5:2020)
Microbiologie de la chaîne alimentaire - Validation des méthodes - Partie 5: Protocole
pour la validation interlaboratoires de méthodes non commerciales par plan factoriel
(ISO 16140-5:2020)
Ta slovenski standard je istoveten z: EN ISO 16140-5:2020
ICS:
07.100.30 Mikrobiologija živil Food microbiology
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

EN ISO 16140-5
EUROPEAN STANDARD
NORME EUROPÉENNE
August 2020
EUROPÄISCHE NORM
ICS 07.100.30
English Version
Microbiology of the food chain - Method validation - Part 5:
Protocol for factorial interlaboratory validation for non-
proprietary methods (ISO 16140-5:2020)
Microbiologie de la chaîne alimentaire - Validation des Mikrobiologie der Lebensmittelkette -
méthodes - Partie 5: Protocole pour la validation Verfahrensvalidierung - Teil 5: Arbeitsvorschrift für
interlaboratoires de méthodes non commerciales par eine faktorielle Ringversuch-Verfahrensvalidierung
plan factoriel (ISO 16140-5:2020) (ISO 16140-5:2020)
This European Standard was approved by CEN on 25 May 2020.

CEN members are bound to comply with the CEN/CENELEC Internal Regulations which stipulate the conditions for giving this
European Standard the status of a national standard without any alteration. Up-to-date lists and bibliographical references
concerning such national standards may be obtained on application to the CEN-CENELEC Management Centre or to any CEN
member.
This European Standard exists in three official versions (English, French, German). A version in any other language made by
translation under the responsibility of a CEN member into its own language and notified to the CEN-CENELEC Management
Centre has the same status as the official versions.

CEN members are the national standards bodies of Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia,
Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway,
Poland, Portugal, Republic of North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Turkey and
United Kingdom.
EUROPEAN COMMITTEE FOR STANDARDIZATION
COMITÉ EUROPÉEN DE NORMALISATION

EUROPÄISCHES KOMITEE FÜR NORMUNG

CEN-CENELEC Management Centre: Rue de la Science 23, B-1040 Brussels
© 2020 CEN All rights of exploitation in any form and by any means reserved Ref. No. EN ISO 16140-5:2020 E
worldwide for CEN national Members.

Contents Page
European foreword . 3

European foreword
This document (EN ISO 16140-5:2020) has been prepared by Technical Committee ISO/TC 34 "Food
products" in collaboration with Technical Committee CEN/TC 463 “Microbiology of the food chain” the
secretariat of which is held by AFNOR.
This European Standard shall be given the status of a national standard, either by publication of an
identical text or by endorsement, at the latest by February 2021, and conflicting national standards
shall be withdrawn at the latest by February 2021.
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. CEN shall not be held responsible for identifying any or all such patent rights.
According to the CEN-CENELEC Internal Regulations, the national standards organizations of the
following countries are bound to implement this European Standard: Austria, Belgium, Bulgaria,
Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland,
Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Republic of
North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Turkey and the
United Kingdom.
Endorsement notice
The text of ISO 16140-5:2020 has been approved by CEN as EN ISO 16140-5:2020 without any
modification.
INTERNATIONAL ISO
STANDARD 16140-5
First edition
2020-07
Microbiology of the food chain —
Method validation —
Part 5:
Protocol for factorial interlaboratory
validation for non-proprietary
methods
Microbiologie de la chaîne alimentaire — Validation des méthodes —
Partie 5: Protocole pour la validation interlaboratoires de méthodes
non commerciales par plan factoriel
Reference number
ISO 16140-5:2020(E)
©
ISO 2020
ISO 16140-5:2020(E)
© ISO 2020
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO 2020 – All rights reserved

ISO 16140-5:2020(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 2
4 General principles for the factorial interlaboratory validation of non-proprietary
methods . 3
4.1 General . 3
4.2 Validation against a reference method . 3
4.3 Validation without a reference method . 3
5 Qualitative methods — Technical protocol for factorial interlaboratory validation .3
5.1 In-house validation study . 3
5.2 Interlaboratory validation study against a reference method . 4
5.2.1 General considerations . 4
5.2.2 Measurement protocol . 4
5.2.3 Selection of the factors to be studied . 5
5.2.4 Experimental design . 6
5.3 Calculations and summary of data . 6
5.4 Interpretation of data . 8
5.4.1 Paired study . 8
5.4.2 Unpaired study . 9
5.4.3 Analysis of factorial effects with respect to RLOD . 9
6 Quantitative methods — Technical protocol for factorial interlaboratory validation .10
6.1 In-house validation study .10
6.2 Interlaboratory validation study against a reference method .10
6.2.1 General considerations .10
6.2.2 Measurement protocol .10
6.2.3 Experimental design .11
6.3 Calculations, summary, and interpretation of data .12
6.3.1 Summary of test results .12
6.3.2 Precision data .13
6.3.3 Accuracy profile .14
6.4 Interlaboratory validation study without a reference method .14
Annex A (informative) List of factors and factor levels for factorial validation.15
Annex B (informative) Example of a factorial interlaboratory study for a quantitative method .17
Annex C (informative) Example of a factorial interlaboratory study for a qualitative method .28
Bibliography .34
ISO 16140-5:2020(E)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www .iso .org/ patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO’s adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/
iso/ foreword .html.
This document was prepared by Technical Committee ISO/TC 34, Food products, Subcommittee SC 9,
Microbiology, in collaboration with the European Committee for Standardization (CEN) Technical
Committee CEN/TC 463, Microbiology of the food chain, in accordance with the Agreement on technical
cooperation between ISO and CEN (Vienna Agreement).
A list of all parts in the ISO 16140 series can be found on the ISO website.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html.
iv © ISO 2020 – All rights reserved

ISO 16140-5:2020(E)
Introduction
0.1  The ISO 16140 series
The ISO 16140 series has been expanded in response to the need for various ways to validate or verify
test methods. It is the successor to ISO 16140:2003. The ISO 16140 series consists of six parts with the
general title, Microbiology of the food chain — Method validation:
— Part 1: Vocabulary;
— Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method;
— Part 3: Protocol for the verification of reference methods and validated alternative methods in a single
laboratory;
— Part 4: Protocol for method validation in a single laboratory;
— Part 5: Protocol for factorial interlaboratory validation for non-proprietary methods;
— Part 6: Protocol for the validation of alternative (proprietary) methods for microbiological confirmation
and typing procedures.
ISO 17468 is a closely linked International Standard, which establishes technical rules for the
development and validation of standardized methods.
In general, two stages are needed before a method can be used in a laboratory.
— The first stage is the validation of the method. Validation is conducted using a study in a single
laboratory followed by an interlaboratory study (see ISO 16140-2, ISO 16140-6, and as described in
this document). In the case when a method is validated within one laboratory (see ISO 16140-4), no
interlaboratory study is conducted.
— The second stage is method verification, where a laboratory demonstrates that it can satisfactorily
perform a validated method. This is described in ISO 16140-3. Verification is only applicable to
methods that have been validated using an interlaboratory study.
In general, two types of methods are distinguished: reference methods and alternative methods.
A reference method is defined in ISO 16140-1:2016, 2.59, as an “internationally recognized and widely
accepted method”. The note to entry clarifies that “these are ISO standards and standards jointly
published by ISO and CEN or other regional/national standards of equivalent standing”.
In the ISO 16140 series, reference methods include standardized reference (ISO and CEN) methods as
defined in ISO 17468:2016, 3.5, as a “reference method described in a standard”.
An alternative method (method submitted for validation) is defined in ISO 16140-1:2016, 2.4, as a
“method of analysis that detects or quantifies, for a given category of products, the same analyte as
is detected or quantified using the corresponding reference method”. The note to entry clarifies that:
“The method can be proprietary. The term ‘alternative’ is used to refer to the entire ‘test procedure
and reaction system’. This term includes all ingredients, whether material or otherwise, required for
implementing the method.”.
ISO 16140-4 addresses validation within a single laboratory. The results are therefore only valid for
the laboratory that conducted the study. In this case, verification (as described in ISO 16140-3) is not
applicable. This document, ISO 16140-5, describes protocols for non-proprietary methods where a
more rapid validation is required or when the method to be validated is highly specialized and the
number of participating laboratories required by ISO 16140-2 cannot be reached. ISO 16140-4 and this
document can be used for validation against a reference method. ISO 16140-4 (regarding qualitative
and quantitative methods) and this document (regarding quantitative methods only) can also be used
for validation without a reference method.
ISO 16140-5:2020(E)
The flow chart in Figure 1 gives an overview of the links between the different parts mentioned above.
It also guides the user in selecting the right part of the ISO 16140 series, taking into account the purpose
of the study and the remarks given above.
Figure 1 — Flow chart for application of the ISO 16140 series
NOTE In this document, the words “category”, “type” and/or “item” are sometimes combined with “(food)”
to improve readability. However, the word “(food)” is interchangeable with “(feed)” and other areas of the food
chain as mentioned in Clause 1.
ISO 16140-6 is somewhat different from the other parts in the ISO 16140 series in that it relates to
a very specific situation where only the confirmation procedure of a method is to be validated [e.g.
the biochemical confirmation of Enterobacteriaceae (see ISO 21528-2)]. The confirmation procedure
advances a suspected (presumptive) result to a confirmed positive result. The validation of alternative
typing techniques (e.g. serotyping of Salmonella) is also covered by ISO 16140-6. The validation study
in ISO 16140-6 clearly defines the selective agar(s) from which strains can be confirmed using the
alternative confirmation method. If successfully validated, the alternative confirmation method can
only be used if strains are recovered on an agar that was used and shown to be acceptable within the
validation study. Figure 2 shows the possibilities where an alternative confirmation method validated
in accordance with ISO 16140-6 can be applied (see text in the boxes).
vi © ISO 2020 – All rights reserved

ISO 16140-5:2020(E)
Figure 2 — Use of validated alternative confirmation methods (see ISO 16140-6)
EXAMPLE An example application of a validated alternative confirmation method is as follows.
An alternative confirmation method based on ELISA has been validated (in accordance with ISO 16140-6) to
replace the biochemical confirmation for Salmonella as described in ISO 6579-1. In the validation study, XLD
(mandatory agar in accordance with ISO 6579-1) plus BGA and a specified chromogenic agar (two optional agars
for second plating in accordance with ISO 6579-1) were used as the agars to start the confirmation. The validated
confirmation method can be used to replace the biochemical confirmation under the following conditions:
— by laboratories using the ISO 6579-1; or
— by laboratories using an ISO 16140-2 validated alternative method that refers to ISO 6579-1 for confirmation; or
— by laboratories using an ISO 16140-2 validated alternative method that starts the confirmation from XLD
and/or BGA agar and/or the specified chromogenic agar.
The validated confirmation method cannot be used under the following conditions:
— by laboratories using an ISO 16140-2 validated alternative method that refers only to agars other than those
included in the validation to start the confirmation (e.g. Hektoen agar and SS agar only); or
— by laboratories using an ISO 16140-2 validated alternative method that refers only to a confirmation
procedure that does not require isolation on agar.
0.2  Validation protocols in the ISO 16140 series
An interlaboratory validation study, in accordance with ISO 16140-2, requires at least eight laboratories
for quantitative methods and at least ten laboratories for qualitative methods.
This document provides a protocol that addresses the special case where the number of laboratories
required in an interlaboratory validation of a method by ISO 16140-2 cannot be achieved. The protocol
allows a method validation based on a minimum of four laboratories. It applies, for example, in situations
where there is an urgent need for a validated method but the in-house and interlaboratory studies in
accordance with ISO 16140-2 take too long to complete. This document also addresses the problem of
method validation of highly specialized methods, for which only a few laboratories might be available
for a validation study. This document can only be used for non-proprietary methods. Table 1 provides
an overview of the different protocols.
ISO 16140-5:2020(E)
Table 1 — Overview of different validation protocols described in the ISO 16140 series
Number of participating
With reference method Without reference method
laboratories
1 ISO 16140-4 ISO 16140-4
4 to 7 (quantitative method)/ This document: for non-proprietary This document: for non-proprietary
4 to 9 (qualitative method) methods only quantitative methods only
≥ 8 (quantitative method)/ ISO 16140-2: for the interlaboratory
Not applicable
≥ 10 (qualitative method) study part
In order to reduce the number of required laboratories to a minimum of four, while maintaining the
applicability of the validation to all laboratories, the protocol is based on a factorial experimental design.
In the factorial design, factors such as the technician or culture medium are altered simultaneously,
and the method is used in a range of different factor settings. This approach allows a more detailed
analysis of the precision parameters of the method while, at the same time, requiring a smaller number
of laboratories and tests.
viii © ISO 2020 – All rights reserved

INTERNATIONAL STANDARD ISO 16140-5:2020(E)
Microbiology of the food chain — Method validation —
Part 5:
Protocol for factorial interlaboratory validation for non-
proprietary methods
1 Scope
This document specifies the general principles and the technical protocols (based on orthogonal,
factorial studies) for the validation of non-proprietary methods for microbiology of the food chain.
This document is applicable to the validation of methods used for the analysis (detection or
quantification) of microorganisms in:
— products intended for human consumption;
— products intended for animal feeding;
— environmental samples in the area of food and feed production, handling;
— samples from the primary production stage.
This document is, in particular, applicable to bacteria and fungi. Some clauses can be applicable to other
(micro)organisms or their metabolites, to be determined on a case-by-case basis.
This document specifies protocols for the validation against a reference method for both quantitative
and qualitative methods. This document also provides a protocol for the validation of quantitative
methods without a reference method. Qualitative methods cannot be validated without a reference
method in accordance with this document.
NOTE ISO 16140-2 specifies the general principle and the technical protocol for the validation of alternative,
mostly proprietary, methods against a reference method.
This document is only applicable to the validation of methods that are fully specified with regard to all
relevant parameters (including tolerances on temperatures and specifications on culture media) and
that have already been optimized.
Methods that have been validated in accordance with this document can be used by the laboratories of
the specified population of laboratories.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO 7218, Microbiology of food and animal feeding stuffs — General requirements and guidance for
microbiological examinations
ISO 16140-1:2016, Microbiology of the food chain — Method validation — Part 1: Vocabulary
ISO 16140-2:2016, Microbiology of the food chain — Method validation — Part 2: Protocol for the validation
of alternative (proprietary) methods against a reference method
ISO 16140-5:2020(E)
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 16140-1 and the following apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
factor
qualitative or quantitative parameter within the method that can be varied at two or more levels within
the limits of the specified method
EXAMPLE Technician.
Note 1 to entry: In this document, only those factors that are in line with the protocol of the method are
considered.
3.2
factor level
value of the factors (3.1) within the experimental design
EXAMPLE Technician “a”, technician “b”, etc.
Note 1 to entry: In this document, each factor is varied at two factor levels: “a” and “b”.
Note 2 to entry: This definition is based on how ISO 3534-3:2013, 3.1.12, defines “factor level”. In ISO 3534-3:2013,
3.1.12, the definition is more general, but the statistical meaning is the same.
3.3
orthogonal design
factorial design, in which for every pair of factors (3.1), each combination of factor levels (3.2) occurs the
same number of times across the possible factor levels
Note 1 to entry: This definition is based on how ISO 3534-3:2013, 3.1.31, defines “orthogonal array”, but for
“orthogonal design”, a more general and more theoretical definition is used.
3.4
reference value
estimated concentration level obtained with the reference method
3.5
reference value
estimated concentration
level obtained from the concentration level of the inoculum
3.6
setting
combination of factor levels (3.2)
EXAMPLE Technician “a” + culture media “b” + temperature “a” + etc.
Note 1 to entry: These conditions can be described by the combination of levels of factors varied within the study.
2 © ISO 2020 – All rights reserved

ISO 16140-5:2020(E)
4 General principles for the factorial interlaboratory validation of non-
proprietary methods
4.1 General
This document uses a protocol based on orthogonal, factorial studies. A high certainty of the determined
method validation parameters is achieved by focusing on suitable factors (e.g. technician, culture media,
sample preparation, temperature, test duration) that can influence the test result. This allows the
number of required laboratories to be reduced to a minimum of four. General concepts and considerations
given in ISO 16140-2 shall apply, unless explicitly excluded. The validation can be conducted against a
reference method or, in the case of a quantitative method, without a reference method.
The outcome of the validation study applies to:
— any laboratory: if the four laboratories can be considered a “random sample” of independent
laboratories from different organizations;
— all laboratories within an organization: if the four laboratories can be considered a “random sample”
of laboratories at different sites from one organization.
4.2 Validation against a reference method
The validation protocol comprises two phases:
— an in-house validation study of the non-proprietary method against the reference method carried
out in the organizing laboratory (see ISO 16140-1:2016, 2.45);
— an interlaboratory study of the non-proprietary method against the reference method carried out
in different laboratories.
The technical protocol for performing the in-house validation study and the interlaboratory study are
given in Clauses 5 and 6, depending upon whether the test method is qualitative or quantitative in nature.
The selected factors for the factorial interlaboratory study should be relevant and applied to both the
reference and the alternative method.
4.3 Validation without a reference method
The validation protocol applies to the validation of quantitative methods only. It comprises two phases:
— an in-house validation study of the non-proprietary method against the reference value carried out
in the organizing laboratory;
— an interlaboratory study of the non-proprietary method against the reference value carried out in
different laboratories.
The technical protocol for performing the in-house validation study and the interlaboratory study for
quantitative methods are given in Clause 6.
5 Qualitative methods — Technical protocol for factorial interlaboratory
validation
5.1 In-house validation study
The in-house validation study is the part of the validation process that is performed in the organizing
laboratory. It can be conducted in accordance with the conventional approach or the factorial approach,
as described in ISO 16140-4.
ISO 16140-5:2020(E)
An in-house validation study can be used to demonstrate the performance of the method for the
laboratory that conducted the study. It is the first step in the framework of general method validation.
The in-house validation study assesses the performance of the method across (food) categories, (food)
types and (food) items, whereas the interlaboratory study assesses the performance of the method
across laboratories.
5.2 Interlaboratory validation study against a reference method
5.2.1 General considerations
The aim of the interlaboratory study is to compare the performance of the reference method to the
alternative method in terms of the RLOD obtained by different laboratories. The results of the same
set of samples, examined under reproducibility conditions, are compared with pre-set criteria for the
acceptable difference between the reference method and the alternative method. Wherever possible,
the study conditions should reflect the normal variation between laboratories.
The interlaboratory study is planned by the organizing laboratory. The organizing laboratory prepares
the samples and a data sheet for the recording of all experimental data and critical experimental
conditions used by each laboratory. Each laboratory shall demonstrate its competence in the use of the
alternative and the reference method in accordance with ISO 7218 prior to participating in the study.
Technicians involved in the preparation of the samples used in the interlaboratory study shall not take
part in the testing of the samples in the interlaboratory study.
5.2.2 Measurement protocol
A minimum of four independent laboratories are required to conduct the test series.
The laboratories shall belong to different organizations or shall be located at different sites.
If all laboratories belong to one organization or network, results of the validation study can only be
used by laboratories belonging to this organization or network.
EXAMPLE 1 Laboratories belonging to one public or private organization.
EXAMPLE 2 Network of federal, state and/or provincial laboratories.
EXAMPLE 3 Network of national reference laboratories coordinated by a European Union reference
laboratory.
If possible, more than four laboratories should participate, so that results from at least eight technicians
are available for analysis even if, for some reason, certain data cannot be used.
The protocol is as follows.
— In cases where different enrichment protocols for the alternative method exist, the most challenging
enrichment protocol shall be selected, e.g. the protocol having the shortest incubation time or the
most selective enrichment conditions. The selected (food) type shall be relevant for the chosen
enrichment protocol. The (food) item, which is used to prepare the test samples, should contain
natural background microbiota.
— The selected (food) item shall be artificially contaminated with the target organism. The protocol
for inoculation of the samples shall be appropriate for the selected (food) item. Samples shall
be prepared by the organizing laboratory to ensure homogeneity between and within samples
using preparation protocols contained in ISO 16140-2:2016, Annex C. In general, liquid samples
(compared to solid samples) give greater assurance of homogeneity. The samples should be shown
to be homogeneous by the organizing laboratory. Homogeneity tests and criteria for acceptance are
described in ISO 22117.
4 © ISO 2020 – All rights reserved

ISO 16140-5:2020(E)
— At least three different levels of contamination shall be used: a blank (L ) and two levels (L and
0 1
L ). Level L should be between the LOD of the reference method (LOD ) and the LOD of the
2 1 50 50,ref 50
alternative method (LOD = RLOD · LOD ). Level L should be 1 log above level L . Level L
50,alt 50,ref 2 10 1 1
shall produce fractional positive results.
— Inocula should be enumerated using a non-selective medium. Enumeration shall be performed as
described in ISO 7218.
— For the selected (food) item, for each setting and for each laboratory, four replicates shall be
conducted at level L One replicate is conducted at the two other levels L and L .
1. 0 2
— According to the factorial, orthogonal design, eight settings shall be conducted by each laboratory
for both test methods (alternative and reference) and for all levels of contamination (L , L and L ).
0 1 2
In total 96 tests (1 + 4 + 1 replicates × 8 settings × 2 methods) are conducted in each laboratory.
— All samples should be blind-coded to ensure that the technicians are not aware of their level of
contamination.
— Settings shall be common to both test methods (paired and unpaired). Paired results are obtained
when the primary enrichment is the same for the alternative and reference method. That is,
one test portion is used to conduct tests according to the alternative and the reference method.
Unpaired results are obtained when the alternative and reference methods use different primary
enrichments. That is, two sets of test portions are required: one test portion is analysed by the
alternative method and another test portion by the reference method.
— The data are reported in two tables, giving the results from the reference method and from the
alternative method before and after confirmation of the results.
— If the results for alternative and reference methods have been obtained from the same initial
enrichment broth (paired data), there is no need to confirm the presumptive results of the
alternative method if the results agree with that obtained with the reference method. However,
confirmation of the positive result is required when the reference method gives a negative
result and the alternative method gives a positive result.
— If the results for alternative and reference methods have been obtained from different
enrichments (unpaired data), then all enrichments obtained with the alternative method shall
be taken forward for confirmation. Confirmation pathways are described in ISO 16140-2:2016,
5.1.3.3.
— The organizing laboratory can indicate that broths, plates and/or isolates shall be retained for a
certain period of time to enable confirmation of results obtained by a laboratory, if needed.
— The analysis of samples shall be performed by each laboratory on the stipulated date.
5.2.3 Selection of the factors to be studied
Decisions on the most suitable factors for the particular study should be based on expert knowledge. For
example, optimal conditions are specified in each method, e.g. incubation temperature and duration at
37° C and 24 h, and these will give the best results. However, ranges around these, due to inaccuracies of
the instrument, but which provide still acceptable conditions (e.g. 37 °C ± 1 °C, 24 h ± 1 h), are permitted
and the study design should test the extremes of these. Acceptable operating conditions for equipment
are described in ISO 7218. Other influences such as stress conditions can also be taken into account.
To estimate the accuracy under routine conditions, relevant method factors that are difficult to control
shall be selected and varied systematically, e.g. technicians, culture media and incubation conditions.
The choice of these factors and factor levels is crucial to the reliability of the test result. For unpaired
study designs, chosen settings biased against the reference method cannot be used.
Five relevant method factors shall be varied simultaneously, each on two levels.
ISO 16140-5:2020(E)
For methods for culturable microorganisms, the factors “technicians” and “culture media” have the
greatest significance and shall be included in all studies as follows.
— Technicians: Tests shall be independently conducted in each of the laboratories by two technicians.
— Culture media: Use culture media from two different manufacturers, if available, or two different
batches of culture media (lots), or pre-prepared versus prepared from dehydrated media. If possible,
each laboratory should use manufacturers/batches that are different from the ones used in the
other laboratories.
Three other factors shall be included in the study. A non-comprehensive list of grouped potential factors
is provided in Annex A. Depending on the method, one factor from each of the most relevant groups
shall be selected.
Factors and factor levels shall reflect the normal variation within routine testing laboratories. Factors
are studied simultaneously using the study design described in 5.2.4.
5.2.4 Experimental design
Each setting is a combination of levels of five method factors. At each setting, each of the laboratories
conducts (1 + 4 + 1) tests at levels L , L and L for both test methods. Eight different settings (1 to 8)
0 1 2
shall be considered. The study design is shown in Table 2, where “a” and “b” represent the levels of the
respective factors.
NOTE This study design (5 factors) is different from the design used in ISO 16140-4 for single-laboratory
validation studies (4 factors).
Table 2 — Study design for qualitative methods to be conducted by each of at least four
laboratories
Factor 2: Factor 3: Factor 4: Factor 5:
Factor 1:
Setting culture e.g. thawing e.g. incubation e.g. background
technician
medium process condition microbiota
1 a a a a
2 a b b b b
3 a a b b
4 b b a a
5 a b a b
6 b b a b a
7 a b b a
8 b a a b
With four laboratories, eight settings, three levels of contamination and two test methods (reference
and alternative) the minimum number of individual tests is 4 × 8 × (1 + 4 + 1) × 2 = 384.
The organizing laboratory shall examine the raw data and other information requested in the data
sheet to ascertain that all laboratories have performed the analyses in accordance with both the
alternative and reference methods as written. When there is evidence that results might be obtained
under inappropriate conditions and/or that the methods have not been strictly followed, these or all
results from the laboratory are excluded for further analysis.
Annex C shows an example of a factorial interlaboratory study for a qualitative method.
5.3 Calculations and summary of data
The results obtained by the individual laboratories in the interlaboratory study are summarized in
Tables 3 and 4.
6 © ISO 2020 – All rights reserved

ISO 16140-5:2020(E)
Table 3 — Fraction of positive results by the reference method
Contamination level
Laboratory
L L L
0 1 2
Laboratory 1 /8 /32 /8
Laboratory 2 /8 /32 /8
Laboratory 3 /8 /32 /8
Etc…. /8 /32 /8
Total
Table 4 — Fraction of positive results (before and after confirmation) by the alternative method
Contamination level
Laboratory L L L
0 1 2
Presumptive Confirmed Presumptive Confirmed Presumptive Confirmed
Laboratory 1 /8 /8 /32 /32 /8 /8
Laboratory 2 /8 /8 /32 /32 /8 /8
Laboratory 3 /8 /8 /32 /32 /8 /8
Etc…. /8 /8 /32 /32 /8 /8
Total
Subsequent calculations, using the data from the fractional recovery level L only, are carried out in
accordance with ISO 16140-2:2016, 5.2.3. These calculations shall be conducted across all factor
settings, 1 to 8, and also separately for each subgroup of factor settings that belongs to one of the five
factors and one of the two factor levels, as shown in Table 5.
ISO 16140-5:2020(E)
Table 5 — Summary of results
a b
Factor levels Settings PA NA ND PD FP N SE SE RT SP SP FPR
(alt) (ref) (alt) (ref)
% % % % % %
All settings 1 to 8
Factor 1 (technician) “a” – see
NOTE 2 1,2,3,4
Factor 1 (technician) “b” – see
NOTE 2 5,6,7,8
Factor 2 (culture medium) “a” 1,3,5,7
Factor 2 (culture medium) “b” 2,4,6,8
Factor 3 (e.g. thawing process) “a” 1,3,6,8
Factor 3 (e.g. thawing process) “b” 2,4,5,7
Factor 4 (e.g. incubation condition)
“a” 1,4,5,8
Factor 4 (e.g. incubation condition)
“b” 2,3,6,7
Factor 5 (e.g. background
microbiota) “a” 1,4,6,7
Factor 5 (e.g. background
microbiota) “b” 2,3,5,8
Key
PA: positive agreement, NA: negative agreement, ND: negative deviation, PD: positive deviation, FP: false positive, N: sum, SE :
...