ISO 16100-4:2006

INTERNATIONAL ISO
STANDARD 16100-4
First edition
2006-12-15

Industrial automation systems and
integration — Manufacturing software
capability profiling for interoperability —
Part 4:
Conformance test methods, criteria and
reports
Systèmes d'automatisation industrielle et intégration — Profil d'aptitude
du logiciel de fabrication pour interopérabilité —
Partie 4: Méthodes d'essai, critères et rapports de conformité



Reference number
ISO 16100-4:2006(E)
©
ISO 2006

---------------------- Page: 1 ----------------------
ISO 16100-4:2006(E)
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.


©  ISO 2006
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland

ii © ISO 2006 – All rights reserved

---------------------- Page: 2 ----------------------
ISO 16100-4:2006(E)
Contents
Foreword .iv
Introduction.v
1 Scope.1
2 Normative references .1
3 Terms and definitions.2
4 Abbreviated terms .5
5 Conformance framework.5
5.1 Conformance testing .5
5.2 Types of UUTs .6
5.3 Conformance test methodology .6
6 Conformance testing process.9
6.1 "Create CSI" activity .9
6.2 "Create ATC" activity .11
6.3 "Create ETC" activity.11
6.4 "Test UUT" activity.11
7 Conformance of UUTs .12
7.1 Conformance of a capability class structure .12

7.2 Conformance of a capability template.14
7.3 Conformance of a capability profile .16

7.4 Conformance of a capability profile matcher.23

Annex A (informative)  Conformance test for a capability profile.24

Annex B (informative)  Conformance test for a type 1 matcher.28
Bibliography.32

© ISO 2006 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO 16100-4:2006(E)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies
(ISO member bodies). The work of preparing International Standards is normally carried out through ISO
technical committees. Each member body interested in a subject for which a technical committee has been
established has the right to be represented on that committee. International organizations, governmental and
non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the International
Electrotechnical Commission (IEC) on all matters of electro-technical standardization.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives.
Draft International Standards adopted by the technical committees are circulated to the member bodies for
voting. Publication as an International Standard requires approval by at least 75 % of the member bodies
casting a vote.
Attention is drawn to the possibility that some of the elements of this part of ISO 16100 may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights.
ISO 16100 was prepared by Technical Committee ISO/TC 184, Industrial automation systems and
integration, Subcommittee SC 5, Architecture, communications and integration frameworks.
ISO 16100 consists of the following parts, under the general title Industrial automation systems and integration
— Manufacturing software capability profiling for interoperability
Part 1: Framework
Part 2: Profiling methodology
Part 3: Interface services, protocols and capability templates
Part 4: Conformance test methods, criteria and reports
The following part is under preparation
Part 5: Methodology for profile matching using multiple capability classes
iv © ISO 2006 – All rights reserved

---------------------- Page: 4 ----------------------
ISO 16100-4:2006(E)
Introduction
The motivation for ISO 16100 stems from the industrial and economic environment noted in the
ISO/TC 184/SC5 strategic plan, in particular:
a) a growing base of vendor-specific solutions;
b) user difficulties in applying standards;
c) a need to move to modular sets of system integration tools; and
d) a recognition that application software and the expertise to apply that software are assets of the
enterprise.
ISO 16100 is an International Standard for the computer-interpretable and human readable representation of a
software capability profile. Its goal is to provide a method to represent the capability of manufacturing software
relative to its role throughout the life cycle of a manufacturing application, independent of a particular system
architecture or implementation platform.
Certain diagrams in this part of ISO 16100 are constructed following UML conventions. Because not all
concepts embodied in these diagrams are explained in the text, some familiarity with UML on the part of the
reader is assumed.

© ISO 2006 – All rights reserved v

---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO 16100-4:2006(E)

Industrial automation systems and integration ― Manufacturing
software capability profiling for interoperability ―
Part 4:

Conformance test methods, criteria and reports
1 Scope
This part of ISO 16100 specifies the test method, the associated test criteria and the statement format used to
evaluate and declare the degree of conformance of an implementation, i.e. a unit under test (UUT), to the
requirements specified in other parts of ISO 16100.
This part of ISO 16100 includes definitions intended to assist a manufacturer or supplier (first party), a user or
purchaser (second party), or an independent body (third party) to perform the assessment for type evaluation.
This part of ISO 16100 contains the following:
― an enumeration of those conformance aspects that can be used to determine whether an implementation
conforms to ISO 16100;
― a definition of the conformance tests and statements used in declaring which aspects are met by an
implementation;
― a description of the aspects to be included in a conformance statement;
― a set of rules to select valid or invalid combinations of aspects when they are combined.
The following topics are not addressed in this part of ISO 16100:
― matters relating to marks or labels of conformance, certificates of conformance or manufacturers’ or
suppliers’ declarations of conformance;
― dates of implementation or allocation of responsibilities to various parties making use of ISO 16100;
― requirements for production, execution or delivery procedures, unless it is impossible to specify
adequately the conforming product, process or service, respectively, without doing so;
― requirements for quality control during production, execution or delivery of the product, process or service,
respectively.
2 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced document
(including any amendments) applies.
ISO 16100-1:2002 Industrial automation systems and integration — Manufacturing software
capability profiling for interoperability — Part 1: Framework
ISO 16100-2:2003 Industrial automation systems and integration — Manufacturing software
capability profiling for interoperability — Part 2: Profiling methodology
© ISO 2006 – All rights reserved 1

---------------------- Page: 6 ----------------------
ISO 16100-4:2006(E)
ISO 16100-3:2005 Industrial automation systems and integration — Manufacturing software
capability profiling for interoperability — Part 3: Interface services, protocols
and capability templates
REC-xml-20000814 Extensible Markup Language (XML) 1.0 Ed. 2 W3C Recommendation
REC-xmlschema-1-20010502 XML Schema Part 1: Structures
REC-xmlschema-2-20010502 XML Schema Part 2: Datatypes
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply. Some of these terms and
definitions have been taken verbatim or were adapted from other standards. In such cases this is indicated in
brackets with the specific part and subclause of the standard given.
3.1
abstract test case
specification, encapsulating at least one test purpose, that is independent of implementation platform,
parameter values, and method
[adapted from ISO 10303-31:1994, 3.2.1]
3.2
abstract test suite
set of abstract test cases

3.3
capability
set of functions and services with a set of criteria for evaluating the performance of a capability
provider
[ISO 16100-1:2002, 3.3]
NOTE This definition differs from that given in ISO 15531-1 and ISO/DIS 19439, where capability is defined as the
quality of being able to perform a given activity. See IEC 62264-1 for a general definition of capability.

3.4
capability class
element within the capability profiling method that represents software unit functionality and behaviour with
regard to the software unit's role in a manufacturing activity
[ISO 16100-2:2003, 3.3]
3.5
capability profiling
selection of a set of offered services defined by a particular interface within a software interoperability
framework
[ISO 16100-1:2002, 3.4]
2 © ISO 2006 – All rights reserved

---------------------- Page: 7 ----------------------
ISO 16100-4:2006(E)
3.6
conformance
conformity
relation between a specification and a real implementation that is realized when any proposition that is true in
the specification is also true in the implementation
EXAMPLE A profile implementation is in conformance with the template specification that is created according to the
rules in ISO 16100.
3.7
conformance point
specific requirement contained in a set of subclauses in ISO 16100 that are used as a basis to generate and
perform a test to determine if an implementation is conformant
3.8
conformance statement
statement that identifies conformance points of a specification and the behaviour that must be satisfied at these
points
[adapted from ISO/IEC 10746-2:1996, 15.1]
3.9
conformance testing
conformity assessment
testing of a candidate product for the existence of specific characteristics required by a standard in order to
determine the extent to which that product is a conforming implementation
[ISO 10303-31:1994, 3.2.22]
3.10
conformance test report
document written at the end of the conformance assessment process, that provides the overall summary of the
conformance of the UUT to the standard for which conformance testing was carried out, and that gives the
details of the testing
[ISO 10303-31:1994, 3.2.23]
3.11
conforming implementation
implementation which satisfies the conformance requirements, consistent with the capabilities stated in the CSI
[adapted from ISO 10303-31:1994, 3.2 24]
3.12
executable test case
implementation of an abstract test case that is platform-dependent and is associated with parameter values
and a specific test method
3.13
executable test suite
set of executable test cases
3.14
falsification testing
test method developed to find errors in the implementation
[adapted from ISO 10303-31:1994, 3.2.32]
© ISO 2006 – All rights reserved 3

---------------------- Page: 8 ----------------------
ISO 16100-4:2006(E)
3.15
interface
abstraction of the behaviour of an object that consists of a subset of the interactions of that object together with
a set of constraints on when they may occur
[ISO 16100-3:2005, 3.3.3]
3.16
manufacturing software unit
class of software resource, consisting of one or more manufacturing software components, performing a
definite function or role within a manufacturing activity while supporting a common information exchange
mechanism with other units
[ISO 16100-1:2002, 3.12]
NOTE A manufacturing software unit can be modeled as a software object using a UML convention.
3.17
matcher
mechanism to compare an offered capability profile with a required capability profile.
[ISO 16100-3:2005, 3.1.6]
3.18
matching level
qualitative measure of how closely a capability profile of an MSU meets the software functional
requirements of a manufacturing activity
[ISO 16100-3:2005, 3.1.7]
3.19
MSU interoperability
capability of a MSU to support a particular usage of an interface specification in exchanging a set of application
information with another MSU
[ISO 16100-3:2005, 3.1.8]
3.20
profile
set of one or more base specifications or sub-profiles or both, and, where applicable, the identification of
chosen classes, conforming subsets, options and parameters of those base specifications, or sub-profiles
necessary to accomplish a particular function, activity, or relationship
[ISO 16100-2:2003, 3.10]
3.21
reference capability class structure
schema representing a hierarchy of capability classes to be used for capability profiling
[ISO 16100-3:2005, 3.1.11]
3.22
template
schema for a manufacturing software capability profile
[ISO 16100-3:2005, 3.1.14]
4 © ISO 2006 – All rights reserved

---------------------- Page: 9 ----------------------
ISO 16100-4:2006(E)
3.23
unit under test
capability profile, capability template, capability class structure or profile matcher being evaluated to determine
if it meets or provides specific characteristics described in ISO 16100
4 Abbreviated terms

ATC Abstract test case
ATG Abstract test group
ATS Abstract test suite
CITI Conformance information for testing the implementation
CSI  Conformance statement for the implementation
ETC Executable test case
ETG Executable test group
ETS Executable test suite
MSU Manufacturing software unit
UML Unified Modeling Language
UUT Unit under test
XIPI eXtra information for platform implementation
XITI eXtra information for testing the implementation
XML eXtensible Markup Language
5 Conformance framework
5.1 Conformance testing
A UUT, such as capability profile, template, reference capability class structure, or profile matcher shall be
called conforming if its externally visible behaviour fulfils specific conformance requirements in this part of ISO
16100.
Conformance testing shall be used to verify if an implementation meets the requirements of a standard or
specification. Conformance testing is a necessary step toward achieving interoperability, but is not a guarantee
for interoperability. Conformance testing provides developers and users the assurance and confidence that the
conforming UUT behaves as expected, performs functions in a known manner, or possesses a prescribed
interface or format.
The basic conformance testing strategy for ISO 16100 shall be falsification testing. Falsification testing subjects
an implementation to various combinations of valid and invalid inputs, and compares the test outputs to the
corresponding expected outputs as defined in the test criteria in order to determine the degree of conformance.
When a test output does not match the expected output, the deduction that the implementation does not
conform to the specification can be made. When the conformance testing output is true, it does not mean
absolute conformance. Falsification testing shall only demonstrate non-conformance. The use of a greater
variety of test inputs can increase the likelihood of conformance.
© ISO 2006 – All rights reserved 5

---------------------- Page: 10 ----------------------
ISO 16100-4:2006(E)
5.2 Types of UUTs
The interoperability for manufacturing software can be realized through the capability profiling method
described in ISO16100-2. The key phases of this capability profiling method both for MSU capability profiling
and required activity capability profiling are as follows:
a) create a capability class structure and register it in the database;
b) search for a capability class structure in the database according to the manufacturing application
requirements;
c) select capability class from the reference capability class structure in the database;
d) create a capability template and register it in the database;
e) search for a capability template in the database corresponding to a capability class;
f) create a capability profile by filling in each field of the template and register it in the database;
g) match a MSU capability profile with a requirement profile using a profile matcher.
Before registering the UUTs in steps (a), (d) and (f), a conformance test associated with the UUT type shall be
performed on the UUT.
The likelihood of interoperability of MSUs will be ensured when their respective capability profiles have been
validated using a capability class structure, a capability class, and a capability profile template that have also
been validated.
The four types of UUTs that shall undergo conformance testing to ensure the likelihood of interoperability are:
― reference capability class structure;
― capability template;
― capability profile;
― capability profile matcher.

5.3 Conformance test methodology
As shown in Figure 1, the following set of activities shall form a conformance testing process:
a) create CSI;
b) create ATC;
c) create ETC;
d) test UUT.
The process shall begin with the creation of a CSI based on analyzing the conformance points and
conformance test criteria contained in ISO 16100.
Adding XITI and a CSI shall result in the creation of an ATC. XITI shall be UUT type-specific and shall include
those items listed in Table 2 for each UUT type.
Each ATC shall be traceable back to a CSI and shall be implemented as a set of ETCs. For a particular test
platform, extra information as listed in Table 3 shall be combined with the set of ETCs corresponding to an ATS
to form an ETS.


6 © ISO 2006 – All rights reserved

---------------------- Page: 11 ----------------------
ISO 16100-4:2006(E)
{Standard and Specification}
Create CSI
: CSI
{XITI}
Create ATC
: ATC
{XIPI}
Get UUT
Create ETC
: UUT
: ETC
Test UUT
: Conformance Test Report


Figure 1 ― Methodology for developing the conformance testing process
© ISO 2006 – All rights reserved 7

---------------------- Page: 12 ----------------------
ISO 16100-4:2006(E)
When an ETS is performed on a particular UUT the test suite result shall be the combined results from each
ETC belonging to this particular ETS. The test result for each ETC shall either be a PASS or a FAIL condition. A
UUT that fails a specific ETC shall imply that the UUT does not conform to the corresponding ATC that maps to
a set of conformance points and associated test criteria. When the execution of at least one ETC belonging to
an ETS results in a FAIL outcome then the execution of the ETS shall be considered as not in full conformance.
For each type of UUT, there are inputs and outputs for all activities of the conformance test. These inputs and
outputs are detailed as follows:
a) "Create CSI" activity ― see Table 1;
b) "Create ATC" activity ― see Table 2;
c) "Create ETC" activity ― see Table 3;
d) "Test UUT" activity ― see Table 4.

Table 1 ― "Create CSI" inputs and outputs
Type of
Capability class Capability Capability Capability
UUT
Input/
structure template profile profile matcher
output
Standard or • Application • Capability class • Template • Set of elements
Input
specification domain ontology in both
• Dictionary and
well-defined
• Dictionary and taxonomy
MSU and
taxonomy
• Path continuity
requirement
profiles
• CSI for • CSI for template • CSI for profile in • CSI for profile
Output
Capability class in Table 7 Table 8 matcher in Table
structure in 9
Table 6

Table 2 ― "Create ATC" inputs and outputs

Type of
Capability class Capability Capability Capability
UUT
Input/
structure template profile profile matcher
output
CITI
• CITI • CITI • CITI • CITI
Input
XITI
• Relative range • Capability class • Template • Template

and depth inside structure
• Dictionary and • Dictionary and
tree
• Dictionary and taxonomy taxonomy
• UML diagrams taxonomy
for activity
classes and
relationships
ATC
• ATC • ATC • ATC • ATC
Output
• ATS • ATS (refer to • ATS (refer to • ATS (refer to
Annex B) Annex A) Annex C)



8 © ISO 2006 – All rights reserved

---------------------- Page: 13 ----------------------
ISO 16100-4:2006(E)
Table 3 ― "Create ETC" inputs and outputs

Type of
Capability class Capability Capability Capability
UUT
Input/
structure template profile profile matcher
output
ATC • ATC • ATC • ATC • ATC
Input
• ATS • ATS • ATS • ATS

XIPI • Platform • Platform • Platform • Platform

• Application • Template • Template • Capability class
domain ontology schema structure
• Dictionary and
taxonomy

• ETC • ETC • ETC • ETC
Output
• ETS • ETS • ETS • ETS

Table 4 ― "Test UUT" inputs and outputs

Type of
Capability class Capability Capability Capability
UUT
Input/
structure template profile profile matcher
output
ETC
• ETC • ETC • ETC • ETC
Input
• ETS • ETS • ETS • ETS
UUT • Reference class • Template file • Profile file • Matcher

structure file
• Varied inputs for • Varied inputs for • Varied inputs for
• Varied inputs for parameters parameters parameters
parameters

• Conformance • Conformance • Conformance • Conformance
Output
test report test report test report test report

6 Conformance testing process
6.1 "Create CSI" activity
6.1.1 Conformance statements for implementation
For each type of UUT a specific CSI set shall be used to identify a set of conformance points at which
conformance tests can be performed using a corresponding set of conformance test criteria. A CSI shall have a
structure as shown in Figure 2.
The CSI is used to have a better understanding for the conformance assessment and to identify the boundaries
of the testing domain. The conformance statement shall be positive or negative. The statement indicating what
should be done is positive, and the statement indicating what should not be done (prohibited) is negative.
© ISO 2006 – All rights reserved 9

---------------------- Page: 14 ----------------------
ISO 16100-4:2006(E)

CSIs

contains

1. * 1. *

conformance test criteria
conformance points

compliant with
compliant with


conformance test
requirements in other parts

of ISO 16100

Figure 2 — Content of CSIs
6.1.2 Types of conformance points
A conformance point is a conformance requirement in a standard or a specification. At this point, the test should
be made to see if the implementation meets a set of conformance criteria. There are four types of conformance
points as shown in Table 5.
               Table 5 ― Types of conformance points
Implemented in
Type Shall be tested ATC result
UUT
A yes yes PASS
B yes yes PASS or FAIL
C optional yes, if implemented PASS
D optional yes, if implemented PASS or FAIL

6.1.3 The CSI schema
Each CSI is a record in the conformance statement table that has the following columns:
a) Conformance set number — a unique identifier for each conformance set which is composed of a set of
logically related conformance points;
b) Conformance point number — a unique identifier for each conformance point;
c) Conformance point description — a short description for the conformance point to be checked;
d) Specification reference — the clause/subclause and ISO 16100 part number from which the conformance
point is derived;
e) Conformance point type — the type of conformance point according to Table 5;
f) Abstract test criteria — an expression of the expected behaviour.
10 © ISO 2006 – All rights reserved

---------------------- Page: 15 ----------------------
ISO 16100-4:2006(E)
6.2 "Create ATC" activity
For each type of UUT a set of ATCs based on a set of UUT-specific CSIs shall be used to test the UUT. Each
ATC shall be related to the other conformance test elements shown in Figure 3 and have the following
elements:
a) UUT type;
b) a conformance point identifier;
c) conformance point set identifier;
d) the test outcome for a conformance point.
Each ATC shall have an individual test purpose to verify and validate a certain UUT behaviour. ATCs shall be
logically grouped into ATGs according to CSI sets. A complete set of ATGs shall form an ATS for a given type
of UUT.
XITI



1.* 1 1.* 1
CSI CSI Set CSIs for UUT


constraints


0.*


1 1 1


1 1 1


1
1.* 1 1.* 1
ATC ATG ATS





XIPI
constraints
1 1
1

1 1 1
1.*

1.* 1 1.* 1
ETC ETG ETS
1


Figure 3 ― Class diagram for conformance testing elements
6.3 "Create ETC" activity
For each type of UUT a set of ETCs based on UUT-specific ATCs shall be used to generate the
platform-specific implementations of the ATCs. Platform-specific factors can include computer hardware,
software, communication networks, and programming languages. ETCs shall form ETGs in the same manner
as the corresponding set of ATCs form ATGs. A complete set of ETGs shall form an ETS for a given type of
UUT, as shown in Figure 3.
6.4 "Test UUT" activity
For each type of UUT an ETS shall be used to conduct the conformance test. Test inputs shall include a UUT
and other UUT-specific information required to perform each ETC belonging to the ETS. Test outputs shall
include a conformance test report having the following elements:
a) a statement of the conformance level that shall contain one of the following values:
1) FULL CONFORMANCE ― when all types of conformance points passed the ETS;
2) MINIMAL CONFORMANCE ― when all type A and C conformance points passed the ETS;
3) NO CONFORMANCE ― when any type A or C conformance point did not pass the ETS.
© ISO 2006 – All rights reserved 11

---------------------- Page: 16 ----------------------
ISO 16100-4:2006(E)
NOTE The statement of the conformance level may not be reliable if the ETS has not been properly derived from the
ATS.
b) corresponding information that shall
...