SIST EN ISO/IEC 9646-5:1997

SLOVENSKI STANDARD
SIST EN ISO/IEC 9646-5:1997
01-december-1997
Information technology - Open Systems Interconnection - Conformance testing
methodology and framework - Part 5: Requirements on test laboratories and
clients for the conformance assessment process (ISO/IEC 9646-5:1994)
Information technology - Open Systems Interconnection - Conformance testing
methodology and framework - Part 5: Requirements on test laboratories and clients for
the conformance assessment process (ISO/IEC 9646-5:1994)
Informationstechnik - Kommunikation offener Systeme - Methodik der
Konformitätsprüfung - Teil 5: Anforderungen an Testzentren und Testteilnehmer für die
Vorbereitung, Durchführung und Bewertung der Konformitätsprüfung (ISO/IEC 9646-
5:1994)
Technologies de l'information - Interconnexion de systemes ouverts - Cadre général et
méthodologie des tests de conformité OSI - Partie 5: Spécifications pour laboratoires
d'essais et clients pour le procédé d'évaluation de conformité (ISO/IEC 9646-5:1994)
Ta slovenski standard je istoveten z: EN ISO/IEC 9646-5:1996
ICS:
35.100.01 Medsebojno povezovanje Open systems
odprtih sistemov na splošno interconnection in general
SIST EN ISO/IEC 9646-5:1997 en
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------

SIST EN ISO/IEC 9646-5:1997

---------------------- Page: 2 ----------------------

SIST EN ISO/IEC 9646-5:1997

---------------------- Page: 3 ----------------------

SIST EN ISO/IEC 9646-5:1997

---------------------- Page: 4 ----------------------

SIST EN ISO/IEC 9646-5:1997
INTERNATIONAL
ISO/IEC
STANDARD
Second edition
1994-l 2-15
Information technology - Open Systems
Interconnection - Conformance testing
methodology and framework -
Part 5:
Requirements on test laboratories and clients
for the conformance assessment process
Technologies de I’informa tion - Interconnexion de syst&mes ouverts -
Cadre g&&al et m&hodologie des tests de conformit OS/ -
Partie 5: Spkifications pour laboratoires d’essais et clients pour le
pro&d6 d’&alua tion de conformit
Reference number
ISO/lEC 9646-5:1994(E)

---------------------- Page: 5 ----------------------

SIST EN ISO/IEC 9646-5:1997
ISO/IEC 9646-5 : 1994 (E)
Contents
Page
Foreword. .i v
.......................................................
Introduction . . V
1 Scope.l .l
.....................................................
2 Nonnativereferences. .
..............................................
3 Definitions .2
......................................................
4 Abbreviations
...................................................... .
5 Overview of the conformance assessment process.
........................ .3
5.1 Introduction. .3
...................................................
5.2 Preparation for testing.
........................................... .3
5.3Testo~erations
............................................... ...4
5.4 Test report production.
.4
...........................................
6 Preparation for testing.
.............................................. .4
6.1 Introduction. .4
...................................................
6.2 Requirements for testability of the System Under Test . .4
6.3 Communication between the test laboratory and the client . .7
6.4 Documentation for conformance assessment . .9
7 Testoperations .ll
..................................................
7.1 Introduction .ll
..................................................
7.2 Static conformance review .ll
.......................................
7.3 Test selection .14
.................................................
7.4 Test parameterization .15
...........................................
7.5 Verification of Test Coordination Procedures .15
........................
7.6Testcampaign.l 6
7.7 Negotiated exits during the test campaign .18
...........................
8 Test report production. .19
.............................................
8.1 Conformance test reports
........................................ .19
8.2 System Conformance Test Report (SCTR).
.......................... .19
8.3 Protocol Conformance Test Report (PCTR). . .20
9 Comp~ance.2 1
9.1 Test laboratory role . .21
9.2 Client role. .21
...................................................
@ ISO/IEC 1994
All rights reserved. Unless otherwise specified, no part of this publication may be
reproduced or utilized in any form or by any means, electronic or mechanical, including
photocopying and microfilm, without permission in writing from the publisher.
ISO/IEC Copyright Office l Case Postale 56 l CH-1211 Geneve 20 l Switzerland
Printed in Switzerland
ii

---------------------- Page: 6 ----------------------

SIST EN ISO/IEC 9646-5:1997
0 ISOAEC ISO/.IEC 9646-5 : 1994 (E)
Annex
..................... .22
A System Conformance Test Report (SCTR) template.
A.1 Introduction.2 2
A.2Conformance.2 2
.22
A.3 SCTR proforma. .
..................... .26
B Protocol Conformance Test Report (PCTR) template
B.l Introduction.2 6
.26
B.2 Conformance. .
.26
B.3 PCTR proforma. .
C Skeleton IXIT Proforma. 30
............................................
C.l Introduction.3 0
30
C.2 IXIT proforma.
................................................
. . .
111

---------------------- Page: 7 ----------------------

SIST EN ISO/IEC 9646-5:1997
ISO/IEC 9646-5: 1994(E) @ ISOAEC
Foreword
IS0 (the International Organization for Standardization) and IEC (the Inter-
national Electrotechnical Commission) form the specialized system for worldwide
standardization. National bodies that are members of IS0 or IEC participate in the
development of International Standards through technical committees established
by the respective organization to deal with particular fields of technical activity.
IS0 and IEC technical committees collaborate in fields of mutual interest. Other
international organizations, governmental and non-governmental, in liaison with
IS0 and IEC, also take part in the work.
In the field of information technology, IS0 and IEC have established a joint
technical committee, ISO/IEC JTC 1. Draft International Standards adopted by the
joint technical committee are circulated to national bodies for voting. Publication
as an International Standard requires approval by at least 75 % of the national
bodies casting a vote.
International Standard ISO/IEC 9646-5 was prepared by Joint Technical
Committee ISO/IEC JTC 1, Information technology, Subcommittee 21, Open
Systems Interconnection, data management and open distributed processing.
This second edition cancels and replaces the first edition (ISOIIEC 9646-5: 1991)
which has been technically revised.
ISO/IEC 9646 consists of the following parts, under the general title Information
technology - Open Systems Interconnection - Conformance testing methodol-
ogy and framework:
- Part I: General concepts
- Part 2: Abstract Test Suite specification
- Part 3: The Tree and Tabular Combined Notation
- Part 4: Test realization
- Part 5: Requirements on test laboratories and clients for the conformance
assessment process
- Part 6: Protocol profile test specification
- Part 7: Implementation conformance statements
Annexes A and B form an integral part of this part of ISO/IEC 9646. Annex C is
for information only.
iv

---------------------- Page: 8 ----------------------

SIST EN ISO/IEC 9646-5:1997
ISO/IEC 9646-5 : 1994 (E)
0 ISO/IEC
Introduction
Conformance testing requires mutual understanding and agreement between the
test laboratory and the client. This part of ISO/IEC 9646 addresses the roles of both
the test laboratory and the client during the conformance assessment process, the
need to reach mutual agreements between them, and the requirements on each of
them.
The conformance assessment process is the most visible process of conformance
testing, where the results of test suite standardization are put to real use. This is also
the stage at which there is potentially the most scope for variations to occur. As this
part of ISO/IEC 9646 is concerned with the relatively formal process of testing
implementations, it is important that the number and nature of such variations is
very limited. One of the major objectives of standardizing the conformance testing
process is to achieve an acceptable and useful degree of comparability of results of
conformance assessments of similar implementations. If this is to be achieved, not
only should the same source of tests be used (i.e. as specified in appropriate stand-
ards), but also the methods of selecting and pararneterizing these tests, and present-
ing their results, should be, to a large extent, the same.
This part of ISO/IEC 9646 addresses the issues which should be taken into account,
by both the test laboratory and the client, if the necessary consistency of conform-
ance assessment is to be achieved. The target audiences for this part of ISO/IEC
9646 are the test laboratories and their clients. The test laboratory is responsible for
conducting the conformance assessment of an OS1 implementation at the request of
a client. Typically, test laboratories are
1) organizations developing or supplying OS1 implementations (first-party test
laboratories);
2) organizations willing to verify OS1 implementations themselves before
using them (second-party test laboratories);
3) organizations, independent of suppliers or users of OS1 implementations,
whose business is the testing of such implementations (third-party test labora-
tories).
Clients may be implementors or suppliers of real open systems or other OS1 sys-
tems, who are applying for their own implementations to be tested. Alternatively,
they may be procurers of those implementations, or any other interested party. The
applicability of this part of ISO/IEC 9646 is independent of the relationship
between the client and the implementation. During the conformance assessment
process, the client is responsible for the conformance statements accompanying the
System Under Test (SUT) and for the configuration of the SUT.

---------------------- Page: 9 ----------------------

SIST EN ISO/IEC 9646-5:1997
ISO/IEC 9646-5 : 1994 (E)
0 ISO/IEC
Secondary, but related, audiences to whom this part of ISO/IEC 9646 could also be
of interest
a) are test realizers;
b) organizations responsible for the accreditation of first-, second- or third-party
test laboratories;
organizations responsible for the issue of test certificates which are based upon
C)
the conformance test reports issued by test laboratories;
d) readers of conformance test reports.
Within this part of ISO/IEC 9646, the conformance assessment process relating to
both the test laboratory and the client is divided into three phases:
i) preparation for testing;
ii) test operation;
iii) production of test reports.
An overview of these three phases is given in clause 5. Clauses 6 to 9 state require-
ments on how to conduct these three phases.
For the purposes of this part of ISO/IEC 9646, it is assumed that a test laboratory is
available and is already organized to provide a conformance assessment service.
The test laboratory is assumed to have acquired from a test realizer (whether or not
the latter belongs to the sarne organization) Means of Testing IUTs, for one or more
OS1 protocols and according to one or more Abstract Test Methods (ATM)s. This
part of ISO/IEC 9646 specifies requirements 011 the test laboratory with respect to
the conduct of the conformance assessment process for a particular client.
Similarly, it is assumed that a client is ready to apply for conformance assessment
of an OS1 implementation. The client is assumed to be familiar with the appropriate
standards, with the concepts of conformance testing and Abstract Test Methods, and
to be ready to cooperate with the test laboratory. This part of ISO/IEC 9646 specifies
requirements on the client with respect to both the testability of the proposed SUT
and the conduct of the conformance assessment process.
It is recommended that this part of ISO/IEC 9646 be read in conjunction with ISO/
IEC 9646- 1.
This part of ISOLEC 9646 is also to be published as ITU-T Recommendation X.294
but not as identical text.

---------------------- Page: 10 ----------------------

SIST EN ISO/IEC 9646-5:1997
INTERNATIONAL STANDARD 0 ISOAEC ISOAEC 9646-S: 1994 (E)
Information technology - Open Systems Interconnection -
Conformance testing methodology and framework - Part 5:
Requirements on test laboratories and clients for the conformance
assessment process
1 Scope
1.1 This part of ISOLEC 9646 specifies requirements on both the test laboratory and the client, for the conduct of the
conformance assessment process. The requirements are those necessary to achieve comparability of results of tests on similar
implementations performed by different test laboratories. Implementations could support one base specification only,
multiple base specifications or one or more profile(s) based on one or more base specifications. This part of ISO/IEC 9646
also provides some guidance on the conformance assessment process.
1.2 The requirements include
a) requirements for the testability of the implementation with respect to Abstract Test Methods;
b) general requirements on the test laboratory and the client applicable to any conformance assessment process;
c) exchange of technical and administrative information, including a System Conformance Statement, an Implementation
Conformance Statement for each relevant base or profile specification, and Implementation extra Information for Testing
for each Abstract Test Suite (ATS) to be used for testing each base specification, combination of base specifications or
component of each profile, as appropriate, plus for profile testing the Profile Test Specification Summary for each profile;
d) cooperation between the test laboratory and the client to reach an agreement on the definition of the Implementation
Under Test), on the Abstract Test Methods and ATSs to be used and on the conditions under which testing will be per-
formed;
e) requirements for the structure and content of the conformance test reports that document the results of the conformance
assessment process.
1.3 This part of ISO/IEC 9646 is applicable equally to those test laboratories which are affiliated to suppliers or procurers,
and those which are independent.
1.4 This part of ISO/IEC 9646 is applicable to conformance assessment of implementations of
- OS1 and Integrated Services Digital Network (ISDN) base specifications that comply with the relevant requirements for
testability in ISO/IEC 9646-2, and/or
- OS1 and ISDN profile specifications that comply with the relevant requirements for testability in ISO/IEC 9646-6,
based on
- conformance testing specifications specified in compliance with ISO/IEC 9646-2,
- and if appropriate based on Profile Test Specification Summaries and Profile Specific Test Specifications specified in
compliance with ISO/IEC 9646-6,
and using Means of Testing (MOT) in compliance with ISO/IEC 9646-4.
1.5 The following are outside the scope of this part of ISO/IEC 9646:
a) the production of diagnostic trace information, additional to that in the conformance log, resulting from testing
performed by the test laboratory, and the supply to the client;
b) aspects of test laboratory operations which are not specific to conformance testing implementations of OS1 base speci-
fications and profiles;
c) accreditation of test laboratories;
d) certification of implementations of OS1 protocols.

---------------------- Page: 11 ----------------------

SIST EN ISO/IEC 9646-5:1997
0 ISO/IEC
ISO/IEC 9646-5: 1994 (E)
2 Normative references
The following standards contain provisions which, through reference in this text, constitute provisions of this part of ISO/IEC
9646. At the time of publication, the editions indicated were valid. All standards are subject to revision, and parties to agreements
based on this part of ISO/IEC 9646 are encouraged to investigate the possibility of applying the most recent editions of the
standards indicated below. Members of IEC and IS0 maintain registers of currently valid International Standards.
IS0 7498: 1984, Information processing systems - Open Systems Interconnection - Basic Reference Model.
(See also CCITT Recommendation X.200-( 1984)).
- Open Systems Interconnection - Service coszventions.
ISO/TR 8509: 1987, Information processing systems
(See also CCITT Recommendation X.210 (1988)).
Open Systems Interconnection - Conformance testing methodology and
ISO/IEC 9646- 1: 1994, Information technology -
flamework - Part I: General concepts.
(See also ITU-T Recommendation X.290 J))
Conformance testing methodology and
ISOIIEC 9646-2: 1994, Informution technology - Open Systems Interconnection -
1
framework - Part 2: Abstract Test Suite specification.
(See also ITU-T Recommendation X.291 J))
ISO/IEC 9646-3: 1992, Information technology - Open Systems Interconnectiolz - Cortformance testing methodology and
flamework - Part 3: The Tree and Tabular Combined Notation (TTCN).
(See also ITU-T Recommendation X.292 (1993))
J) Ir2formation technology - Open Systems Interconnection - Conformance testing methodology and
ISO/IEC 9646-3 Amd 1: ,
.
framework - Part 3: The Tree and Tabular Combined Notation - Amendment 1: ?TCN extensions.
ISO/IEC 9646-4: 1994, Information technology - Opel2 Systems Intercom2ectiorz - Col-lformarzce testing methodology and
-
Famework - Part 4: Test realization.
(See also ITU-T Recommendation X.293 J))
ISO/IEC 9646-6: 1994, Information technology - Open Systems Interconnection - Conformance testing methodology and
f?amework - Part 6: Protocol profile test specification.
(See also ITU-T Recommendation X.295 J)
- Conformance testing methodology andframework
ISO/IEC 9646-7: J), Information technology - Open Systems Interconnection
- Part 7: Implementation Conformance Statements.
(See also ITU-T Recommendation X.296 J))
3 Definitions
For the purposes of this part of ISO/IEC 9646, all the definitions given in ISO/IEC 9646-l apply. The definitions in this clause
also apply to this part of ISO/IEC 9646.
3.1 client checklist: A record of test-related information supplied to the test laboratory by the client during the test preparation
phase of the conformance assessment process.
3.2 client test manager: The person identified by the client organization as being responsible for all matters relating to the
confomlance testing of the IUT.
3.3 negotiated exit (from the conformance assessment process): A poirlt in time at which the test laboratory alld the client
can mutually decide to terminate the conformance assessment process.
being
3.4 operator of the system under test [SUT operator]: The person or persons designated by the client orgallizatioll as
responsible for operation of the SUT during coIlformame testing.
3.5 test laboratory checklist: A record of test-related information supplied to the client by the test laboratory during the test
preparation phase of the conformance assessment process.
3.6 test laboratory manager: The person identified by the test laboratory as being responsible for all matters relating to test
laboratory operations.
3.7 test operator: The person or persons designated by the test laboratory as being responsible for running conformance tests
against the IUT.
1) To be published.
2

---------------------- Page: 12 ----------------------

SIST EN ISO/IEC 9646-5:1997
0 ISO/IEC ISOAEC 9646-5: 1994 (E)
4 Abbreviations
For the purposes of this part of ISO/IEC 9646, the following abbreviations defined in ISO/IEC 9646-l: 1994, clause 4 apply:
ASP: abstract service primitive
ATM: abstract test method
ATS: abstract test suite
basic interconnection test
BIT:
ETS: executable test suite
implementation conformance statement
ICS:
IUT: implementation under test
implementation extra information for testing
IXIT:
MOT: means of testing
MPyT: multi-party testing
OSI: open systems interconnection
point of control and observation
PCO:
PCTR: protocol conformance test report
PETS: parameterized executable test suite
protocol implementation conformance statement
PIGS:
PIXIT: protocol implementation extra information for testing
profile specific test specification
PSTS:
profile test specification
PTS:
RL: requirements list
SATS: selected abstract test suite
scs: system conformance statement
system conformance test report
SCTR:
SPyT: single-party testing
SUT: system under test
TCP: test coordination procedures
TMP: test management protocol
TSS&TP: test suite structure and test purposes
TTCN: tree and tabular combined notation
XRL: IXIT requirements list
5 Overview of the conformance assessment process
5.1 Introduction
Figure 1 of ISO/IEC 9646-l illustrates the conformance assessment process. ISO/IEC 9646-1, subclause 6.3 provides an
overview of the conformance assessment process. An overview of the three phases (preparation for testing, test operations
and test report production) is given as guidance in the following subclauses.
5.2 Preparation for testing
The preparatory phase includes
a) general administrative steps, such as the application by the client, the provision of documents by the test laboratory
describing the general policy, terms and conditions to be followed during test operations, and the provision of information
about the System Under Test (SUT) by the client;

---------------------- Page: 13 ----------------------

SIST EN ISO/IEC 9646-5:1997
ISO/IEC 9646-5: 1994 (E) 0 ISOIIEC
b) checking the completeness of the documents provided by the test laboratory (~~~plenlentation extra Information for Testing
(IXIT) proforma(s)), of those provided by the client (Implementation Conformance Statements(s) (ICS(s)), IXIT(s), System
Conformance Statement (SCS), and also any other information exchanged between the test laboratory and the client; in partic-
ular, the preparatory phase for a profile also includes checking completeness of the profile related documents provided by the
test laboratory (profile IXIT proforma), and those provided by the client (profile ICS and profile IXIT, SCS);
c) analysis of the configuration of the SLIT and choice of a conformance testing specification(s) for the base specification(s)
and profiles, if any, to be tested, either agreeing that the SUT and the test laboratory’s Means of Testing (MOT) are both capa-
ble of supporting the test method(s), or using a negotiated exit if an agreement cannot be reached;
d) preparation of the SUT and the MOT for the testing configuration that results from the choice of test method(s).
Requirements for test preparation, on both the test laboratory and the client, are given in clause 6.
5.3 Test operations
During the second phase, the test operations are carried out. These include
a) the static conformance review, during which detailed analysis of the ICS(s) and IXIT(s) takes place;
b) test selection and parameterization, applied to the executable (or abstract) test suite(s); this determines the Parameterized
Executable Test Suite(s) (PETS(s)) that will be executed;
c) one or more test campaigns, running:
1) Basic Interconnection Tests (optional);
2) capability tests;
3) behaviour tests.
If difficulties are encountered during test operations, it is possible for the client and test laboratory to uegotiate a repetition of a
test campaign as a whole or in part. Alternatively, they can take a negotiated exit from the conformance assessment process.
NOTE - The reasons for the negotiated exit should be are documented in an informal test report.
Requirements for test operations, on both the test laboratory and the client, are given in clause 7.
5.4 Test report production
The third phase, which may begin before the test operations are complete, involves the presentation of the results of the
conformance assessment process. These results are recorded in Protocol Conformance Test Reports (PCTRs) with a summary in
a System Conformance Test Report (SCTR). Requirements for proformas for these test reports are specified in annexes B and A
respectively.
Requirements for test report production are given in clause 8.
6 Preparation for testing
6.1 Introduction
This clause specifies requirements for test preparation on both the test laboratory and the client. Figure 1 illustrates the
preparatory phase of the conformance assessment process. During this phase, both parties ensure that the required documentation
(including SCS, ICS(s) and IXIT(s)) is completed to their mutual satisfaction. Furthermore, checklists are provided for both
parties to help them ensure that they have all the information they need prior to beginning the test operation phase. Each party
shall provide the other with all the information indicated in their own checklist. In particular, the characteristics of the SUT which
determine its configurations and affect the choice of test method(s) shall have been precisely defined. It is assumed that the
requirements for SUT testability have been met by the client before approaching the test laboratory.
As a prerequisite to test operations, the client and the test laboratory agree on the test method(s) and the conditions for the test
campaign. If an agreement is reached, the test laboratory will select the MOT for the chosen test method, and proceed to the test
operations phase; otherwise, a negotiated exit may be taken.
6.2 Requirements for testability of the System Under Test
6.2.1 Client role
6.2.1.1 General
The client shall ensure that the SUT is testable using at least one test method for each base specification implemented in the
Implementation Under Test (IUT).
4

---------------------- Page: 14 ----------------------

SIST EN ISO/IEC 9646-5:1997
ISO/IEC 9646-5: 1994 (E)
0 ISO/IEC
Client Application
SUT Configuration
I
+, Client completes documents
and Testability
++
Client and
Test Laboratory
Test Laboratory
+ Checklist
agree on
test method(s) and ATS(s)
Test Laboratory
IXIT Proforma (s)
Test Laboratory
Client
identifies MOT
completes
and completes
I
IXIT( s)
IXIT proforma( s)
MOT
SUT
Preparation
Preparation
Key ,. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .~
I.
111) Control Flow
1
f-~ Data Flow
Test operations (refer to figures 2, 3 & 5)
Decision Point
0
NE
Negotiated Exit
I
Figure 1 - Preparation for Testing
5

---------------------- Page: 15 ----------------------

SIST EN ISO/IEC 9646-5:1997
0 ISO/IEC
ISO/IEC 9646-5: 1994 (E)
If the IUT includes multi-party protocol(s), then the client shall ensure that the SUT is testable using appropriate Multi-Party
Testing (MPyT) test methods using the appropriate numbers of Lower Testers.
NOTE - This part of ISO/IEC 9646 does not constrain the client to agree to any particular test method(s), as long as at least one test method is
made possible by appropriately configuring the SUT.
Each of the Abstract Test Methods described in ISO/IEC 9646-2: 1994, clause 11, imposes particular requirements on the SUT
with respect to its testability. The requirements vary according to the test method.
For each test method by which the client claims the SUT can be tested, the client shall ensure that the SUT provides the necessary
means of control and observation and that it can enable the appropriate Test Coordination Procedures (TCP) to be performed.
Further requirements for SUT testability for each of the Single-Party Testing (SPyT) Abstract Test Methods are stated below.
The MPyT test methods may impose appropriate combinations of the requirements for the SPyT test methods, depending on the
nature of the Upper Tester(s) and TCP used in each MPyT test method.
6.2.1.2 Local test method
6.2.1.2.1 For the Local test method, the client shall ensure that the TUT upper interface is realized in hardware and can be con-
nected to a test system.
NOTE - The only requirements are that the interface is standardized and that there is a well-defined mapping between the relevant ASPS and
the hardware interface.
6.2.1.2.2 There are no requirements on the client for the TCP.
6.2.1.3 Distributed test method
6.2.1.3.1 For the Distributed test method, the client shall ensure that the SUT contains the means of generating, and indicating
the receipt of, the Abstract Service Primitives (ASPS) for the relevant Points of Control and Observation (PCOs), that are appro-
priate to the envisaged IUT.
NOTES
1 The on means be real ized within the SUT for the control and observation of the effects
ly requirements are that of the relevant ASPS and that
IUT is
the upper service boundary of the either a human-user interface or a standardized programming language interface.
2 Any realization will satisfy this requirement provided
...