EN ISO/IEC 9646-2:1996

SLOVENSKI STANDARD
01-december-1997
Information technology - Open Systems Interconnection - Conformance testing
methodology and framework - Part 2: Abstract Test Suite specification (ISO/IEC
9646-2:1994)
Information technology - Open Systems Interconnection - Conformance testing
methodology and framework - Part 2: Abstract Test Suite specification (ISO/IEC 9646-
2:1994)
Informationstechnik - Kommunikation offener Systeme - Methodik der
Konformitätsprüfung - Teil 2: Spezifikation der Abstrakten Prüfreihe (ISO/IEC 9646-
2:1994)
Technologies de l'information - Interconnexion de systemes ouverts - Cadre général et
méthodologie des tests de conformité OSI - Partie 2: Spécification des suites de tests
abstraites (ISO/IEC 9646-2:1994)
Ta slovenski standard je istoveten z: EN ISO/IEC 9646-2:1996
ICS:
35.100.01 Medsebojno povezovanje Open systems
odprtih sistemov na splošno interconnection in general
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

INTERNATIONAL
ISO~IEC
STANDARD
9646-2
Second edition
1994-l 2-l 5
Information technology - Open Systems
Interconnection
- Conformance testing
methodology and framework -
Part 2:
Abstract Test Suite specification
Technologies de /‘information - lnterconnexion de syst&mes ouverts -
Cadre g6n&al et mkthodologie des tests de conform@ OS/ -
Partie 2: Spkifica tion des suites de tests abstraites
Reference number
lSO/IEC 9646-2:1994(E)
ISO/IEC 9646-2 : 1994 (E)
Contents
Page
..i v
Foreword. .
.
Introduction. .
..l
1 Scope .
...............................................
2 Normative references .l
3 Definitions . .2
4 Abbreviations . .2
5 Compliance . .
6 Conformance requirements in OS1 base specifications . .3
...3
6.1 Itltroduction .
General requirements. . .3
6.2
.......................................... .3
6.3 Conformance clauses
................................. .4
6.4 Multi-specification dependencies
...................................... .4
7 Requirements on ICS proformas
8 Abstract Test Suite production process leading to conformance testing
...4
specifications .
............................ .
9 Conformance requirements and ICS proforma
........................ 5
10 Test Suite Structure and Test Purposes (TSS&TP)
............................................ .5
10.1 Basic requirements.
............................. .5
10.2 Specification of the test suite structure
................................. .7
10.3 Specification of the test purposes
.
10.4 Coverage. .
..................................... .9
10.5 TSS&TP compliance clause
........................................ .9
11 Abstract testing methodology.
...9
11.1 IIltroduction .
............. .lO
11.2 General specification of the Single Party Testing context.
.lO
11.2.1 Introduction. .
.lO
11.2.2 Requirements on the Lower Tester. .
.ll
11.2.3 Requirements on the UpperTester .
Abstract Test Methods for Single Party Testing methods . .l 1
11.3
........................................... .ll
11.3.1 Introduction.
11.3.3 The Distributed test method . .ll
............................... .13
11.3.4 The Coordinated test method
................................... .13
11.3.5 TheRemote test method
0 ISO/IEC 1994
no part of this publication may be
All rights reserved. Unless otherwise specified,
reproduced or utilized in any form or by any means, electronic or mechanical, including
photocopying and microfiIm, without permission in writing from the publisher.
ISO/IEC Copyright Office l Case Postale 56 l CH-1211 Geneve 20 l Switzerland
Printed in Switzerland
ii
0 ISO/IEC ISO/IEC 9646-2 : 1994 (E)
..........................................
11.4 Test method variants.
11.4.1 Embedded and Non-Embedded variants of test methods in the Single
......................................
Party Testing context1
........................................
11.4.2 Multi-user variants
..............
11.5 General specification of the Multi-Party Testing context.
.............................................
11.51 Introduction
..............................
11.52 Lower Tester Control Function
............................................
11.5.3 Upper Testers
...............................
11.5.4 Test Coordination Procedures
...... 15
11.5.5 Illustration of Abstract Test Methods for Multi-Party Testing
.................................
11.6 Choice of Abstract Test Method.
.............................................
11.6.1 Introduction
..............................
11.6.2 Comprehensive testing service
.........................
11.6.3 Types of Implementation Under Test
....................
11.6.4 Applicability of the Abstract Test Methods.
..................................
12 Specification of Abstract Test Suites
12.1 General.19
................ 19
12.2 Use of Tree and Tabular Combined Notation (TTCN).
................................
12.3 Specification of abstract test cases
.......................................
.21
12.4 Assignment Of Verdicts
................ .21
12.5 Abstract Test Suite specification conformance clause
.............................. 21
12.6 Consistency with base specification.
12.7 Copyright.2 1
....................
.21
13 Specification of a Test Management Protocol (TMP)
............ .22
14 Information in an ATS specification concerning use of the ATS
......................
.22
15 Maintenance of Abstract Test Suite specifications
Annex
.......................
.23
A Applicability of the test methods to OS1 protocols
A.lThePhysicallayer.2 3
.....................
.23
A.2 Data Link and Media Access Control protocols.
............................................
.23
A.3 Network protocols.
............................................
.24
A.4 Transport protocol.
..............................................
.25
A.5 Session protocol
............................ .25
A.6 Presentation and Application protocols
.......................................
.27
A.7 Connectionless protocols
........... .28
B Guidance for protocol specifiers to facilitate conformance testing
.................................................
.28
B.l Introduction.
............................................
.28
B.2 Guidance on scope
................................
.29
B.3 Guidance on normative references
............................
.29
B.4 Guidance on requirements and options.
................................
.30
B.5 Checklist for conformance clauses
............................................
.30
B.6 Guidance on PDUs
............................................
.31
B.7 Guidance on states
..3 l
B.8 GuidanceonFDTs.
.........................................
........................................
.32
B.9 Miscellaneous guidance
...... 33
C Relationship between ISO/IEC 9646 and ISO/IEC 7498 service notation
. . .
ISO/IEC 9646-2: 1994(E) OISOAEC
Foreword
IS0 (the International Organization for Standardization) and IEC (the Inter-
national Electrotechnical Commission) form the specialized system for worldwide
standardization. National bodies that are members of IS0 or IEC participate in the
development of International Standards through technical committees established
by the respective organization to deal with particular fields of technical activity.
IS0 and IEC technical committees collaborate in fields of mutual interest. Other
international organizations, governmental and non-governmental, in liaison with
IS0 and IEC, also take part in the work.
In the field of information technology, IS0 and IEC have established a joint
technical committee, ISO/IEC JTC 1. Draft International Standards adopted by the
joint technical committee are circulated to national bodies for voting. Publication
as an International Standard requires approval by at least 75 % of the national
bodies casting a vote.
International Standard ISO/IEC 9646-2 was prepared by Joint Technical
Committee ISO/IEC JTC 1, Information technozogy, Subcommittee 21, Open
Systems Interconnection, data management and open distributed processing.
This second edition cancels and replaces the first edition (ISO/IEC 9646-2: 1991)
which has been technically revised.
ISO/IEC 9646 consists of the following parts, under the general title Information
Open Systems Interconnection - Conformance testing methodol-
technology -
ogy and framework:
- Part 1: General concepts
- Part 2: Abstract Test Suite specification
- Part 3: The Tree and Tabular Combined Notation
- Part 4: Test realization
- Part 5: Requirements on test laboratories and clients for the conformance
assessment process
- Part 6: Protocol profile test specification
- Part 7: Implementation conformance statements
Annexes A, B and C of this part of ISO/IEC 9646 are for information only.
iv
OISO/IEC ISO/IEC 9646-2 : 1994 (E)
Introduction
This part of ISO/IEC 9646 provides a common approach to the specification of OS1
conformance test suites at a level which is independent of the means of executing
those test suites (hereafter called “Abstract Test Suites”). This level of abstraction
is suitable for standardization and facilitates the comparison of results produced by
different organizations which run the corresponding Executable Test Suites.
Clauses 6 and 7 recall that there are requirements on OS1 protocol specifiers which
have to be fulfilled before there can be an objective basis for the process of
The need is expressed for consistent
developing an Abstract Test Suite.
conformance clauses and for ICS proformas in relevant base specifications (e.g.
International Standards or ITU-T Recommendations which specify OS1 protocol
standards).
Clauses 8 to 14 describe the process of developing an Abstract Test Suite, including
the design criteria to be used and guidance on its structure and coverage. The
possible Abstract Test Methods are defined and guidance is given to help the test
suite specifier to decide which test method(s) to use in the production of a particular
test suite. Requirements and guidance are given on the specification of abstract test
cases. These include the subdivision of test cases into test steps and the assignment
of test verdicts to test outcomes.
The test suite specifier is also required to provide information to the test realizers
e. . limitations governing test case selection).
( g
Finally, in clause 15, guidance and requirements are given on test suite
maintenance.
This part of ISO/IEC 9646 is also to be published by ITU-T as Recommendation
X.291.
V
ISO/IEC 9646-2 : 1994 (E) OISOAEC

INTERNATIONAL STANDARD 0 ISOIIEC
ISOLIEC 9646-2: 1994 (E)
Information technology - Open Systems Interconnection -
Conformance testing methodology and framework - Part 2:
Abstract Test Suite specification
1 Scope
1.1 This part of ISO/IEC 9646 specifies the requirements and gives guidance for the production of system-independent
conformance test suites for one or more OS1 specifications. In particular, it is applicable to the production of all OS1
conformance testing specifications including all draft versions of such conformance testing specifications.
1.2 This part of ISO/IEC 9646 is applicable to the production of abstract test cases which check the conformance of an
implementation to the relevant static and/or dynamic conformance requirements by controlling and observing protocol
behaviour. The Abstract Test Methods included in this part of ISO/IEC 9646 are, in fact, capable of being used to specify any
test case which can be expressed abstractly in terms of control and observation of Protocol Data Units (PDUs) and Abstract
Service Primitives (ASPS). Nevertheless, for some protocols, test cases may be needed which cannot be expressed in these
terms. The specification of such test cases is outside the scope of this part of ISO/IEC 9646, although the test cases may
themselves need to be included in a conformance testing specification.
NOTE - For example, some static conformance requirements related to an Application service may require testing techniques which are
specific to that particular Application.
This part of ISO/IEC 9646 is applicable to the production of test suites for testing implementations of one or more adjacent
protocols, whether or not these are embedded under other protocols.
1.3 The following are outside the scope of this part of ISO/IEC 9646:
a) the relationship between Abstract Test Suite (ATS) specification and Formal Description Techniques;
b) testing by means of test methods which are specific to particular applications, protocols or systems, including testing by
means other than PDU exchange.
NOTE - This part of ISO/IEC 9646 applies fully to some but not all Physical layer protocols. Nevertheless, many of the concepts apply to
all protocols.
2 Normative references
The following standards contain provisions which, through reference in this text, constitute provisions of this part of ISO/IEC
9646. At the time of publication, the editions indicated were valid. All standards are subject to revision, and parties to
agreements based on this part of ISO/IEC 9646 are encouraged to investigate the possibility of applying the most recent
editions of the standards listed below. Members of IEC and IS0 maintain registers of currently valid International Standards.
IS0 7498: 1984, Informution processing systems - Open systems - Basic Reference Model.
*
(See also CCITT Recommendation X.200 (1984))
Open Systems Interconnection - Service conventions.
ISOfI’R 8509: 1987, Informution processing systems -
*
(See also CCITT Recommendation X.210 (1988))
Open Systems Intercolznectior2 - Specification of Basic Encoding Rules for
ISO/IEC 8825: 1990, Information technology - _
Abstract Syntax Notation One (ASN.1).
(See also CCITT Recommendation X.209 (1988))
Open Systems Interconnection - Conformance testing methodology and
ISO/IEC 9646-l: 1994, Information technology -
flamework - Part 1: General concepts.
(See also ITU-T Recommendation X.290 J))
Conformance testing methodology and
ISO/IEC 9646-3: 1992, I@ormution technology - Open Systems Intercozznection -
fi-amework - Part 3: The Tree and Tabular Combined Notation (77K’N).
(See also ITU-T Recommendation X.292 (1993))
1) To be published.
0 ISO/IEC
ISO/IEC 9646-2: 1994 (E)
Conformance testing methodology and
ISO/IEC 9646-3 Amd 1: J) , Information technology - Open Systems Interconnection -
framework - Part 3: The Tree and Tabular Combined Notation - Amendment I: ?TCN extensions
Conformance testing methodology and
ISOIIEC 9646- 6: 1994, Information technology - Open Systems Interconnection -
framework - Part 6: Protocol profile test specification.
(See also ITU-T Recommendation X.295 J))
Open Systems Interconnection - Conformance testing methodology and
ISO/IEC 9646- 7: J), Information technology -
framework - Part 7: Implementation Conformance Statements.-
(See also ITU-T Recommendation X.296 -I))
3 Definitions
For the purposes of this part of ISODEC 9646, all the definitions given in ISO/IEC 9646-l apply.
4 Abbreviations
For the purposes of this part of ISO/IEC 9646, the abbreviations given in ISO/IEC 9646-l apply. The following abbreviations
also apply to this part of ISO/IEC 9646.
ACSE: association control service element
ASE: application-service-element
ASN.1: abstract syntax notation one
ASP abstract service primitive
ATM abstract test method
ATS: abstract test suite
FDT formal description techniques
implementation conformance statement
ICS:
implementation under test
IUT:
implementation extra information for testing
IXIT:
lower tester
LT:
LTCF: lower tester control function
MPyT: multi-party testing
OSI: open systems interconnection
PCO: point of control and observation
PCTR: protocol conformance test report
PDU: protocol data unit
RTS: remote transfer service
SAP: service-access-point
single-party testing
SPyT:
system under test
SUT:
test coordination procedures
TCP=
TM-PDU: test management PDU
test management protocol
TMP:
test suite structure and test purposes
TSS&TP:
TTCN: tree and tabular combined notation
.
UT . upper tester
1) To be published.
0 ISO/IEC
ISO/IEC 9646-2: 1994 (E)
5 Compliance
5.1 A base specification which complies with this part of ISO/IEC 9646 shall satisfy all the requirements stated in clauses 6 and 7.
NOTE - Such compliance is a precondition for the base specification to be an effective basis for conformance testing of implementations.
5.2 An Abstract Test Suite (ATS) specification which complies with this part of ISO/IEC 9646
a) shall be a conformance test suite;
b) shall be specified in a test notation standardized by ISOKEC or ITU-T;
c) shall satisfy all the requirements stated in clauses 9 to 15 inclusive;
d) shall be within an ISO/IEC or ITU-T published specification or, in the absence of such a specification, shall be within a
publicly available specification which is in the process of being standardized within ISO/IEC or ITU-T, which has the highest
standardization status available, and which has the status of at least a Committee Draft or equivalent.
NOTE - ATSs outside the standardization process need to be submitted for international standardization before they can fully comply with this
part of ISO/IEC 9646, in order to ensure that they are subject to public scrutiny, correction and acceptance, internationally.
5.3 It is recommended that the test notation used be the Tree and Tabular Combined Notation (TI’CN). If TTCN is used, the ATS
shall comply with ISOIIEC 9646-3.
NOTE - X.290 (1988) is considered to be obsolete for this purpose.
6 Conformance requirements in OS1 base specifications
6.1 Introduction
The meaning of conformance in OS1 is discussed in ISO/IEC 9646-l. It is necessary that there be an unambiguous and objective
understanding of the conformance requirements of an OS1 base specification, as a prerequisite to the production of an ATS for
that specification. Clauses 6 and 7 state the requirements on the relevant specifiers to ensure that there is such an understanding
of the conformance requirements.
Additional guidance is given in annex B.
6.2 General requirements
6.2.1 A clear distinction shall be made between static and dynamic conformance requirements. To avoid ambiguity, they should
be stated separately from one another.
6.2.2 It shall be clear what conformance to the specification means, in the sense of what is required to be done (i.e. mandatory),
what is permitted but not mandatory (i.e. optional), and what is required not be done (i.e. prohibited), in order to conform to it.
6.2.3 It shall always be decidable whether an instance of communication conforms dynamically or not.
For example, it should be possible to look at a record of Protocol Data Unit (PDU) activity and decide whether or not it is valid
with respect to the relevant specification.
6.3 Conformance clauses
6.3.1 Each base specification, which specifies an OS1 protocol, abstract syntax, encoding rules, or information object, shall
include a conformance clause, which shall be expressed clearly and unambiguously.
6.3.2 Conformance clauses shall distinguish between the following categories of infomution:
a) references to clauses which state dynamic conformance requirements;
b) static conformance requirements concerning the implementation of the base specification itself;
c) static conformance requirements concerning multi-specification dependencies (see 6.4).
6.3.3 The requirement to produce an Implementation Conformance Statement (ICS), in compliance with the ICS proforma, shall
be stated separately from the requirements on the implementation of the specification itself.
6.3.4 The conformance clause of a protocol specification should also include
a) the requirement to be able to accept all correct sequences of PDUs received from peers, and respond with correct PDU
sequences;
b) the requirement to be able to respond correctly to all incorrect sequences of PDUs received;
the initiation
in connection oriented protocols, the option to support either of a connection or the acceptance of a connection,
C)
or both;
0 ISO/IEC
ISO/IEC 9646-2: 1994 (E)
d) in connectionless protocols, the option to support the transmission of a PDU, the receipt of a PDU, or both.
6.3.5 A checklist for what should be included or referenced in each conformance clause is given in annex B, B.5.
6.4 Multi-specification dependencies
Multi-specification dependencies may be specified by each base specification requiring the provision of non-mandatory features
in one or more underlying base specifications. If multi-specification dependencies are to be included in an ICS proforma, the
ICS proforma shall merely reflect the multi-specification dependencies specified in the conformance clause of the
corresponding base specification.
Multi-specification dependencies should usually be specified in terms of what elements of a given underlying service are
required in order to support the given protocol or information object. In addition, each underlying protocol specification should
specify which units of the protocol are required if a given element of service can be said to be supported. This refers to the
functionality implied by the element of service, and does not in any way imply the existence of a service interface.
NOTE - This is not conformance to service, but rather is an expression of the conditional requirements that result from the compliance of a
protocol to its service definition.
In cases where it is not possible to express dependencies through the underlying service, they can be specified in terms of the
units of the underlying protocol or other specification req uired to support the higher protocol (the referencing specification).
only be specified in a
Multi-specification dependencie fs should protocol specificat ion if they are needed to preserve the integrity
of that protocol. They should be avoided where they are really defining a profile.
Multi-specification dependencies may also be specified in a similar way in infomlation object specifications.
7 Requirements on ICS proformas
7.1 The specific requirements to be met by suppliers in respect of each ICS they are to provide, shall be stated in the relevant
base specification. The specification of these requirements shall include an ICS proforma. The ICS proforma shall be in the form
of a questionnaire to be completed by the supplier or implementor of an implementation of the relevant base specification.
7.2 The ICS proforma shall cover all major mandatory capabilities, all optional and conditional functions, elements of
procedure, parameters, options, PDUs, timers, multi-specification dependencies and other capabilities identified in the base
specification.
7.3 There shall be a well-defined mapping (by references) from the ICS proforma to the static conformance requirements. The
expression of the static conformance requirements in the ICS proforma shall be consistent with the confolmance clause of the
base specification.
7.4 ISO/IEC 9646-7 provides requirements and guidance on the production of ICS proformas.
8 Abstract Test Suite production process leading to conformance testing specifications
8.1 In order to present the requirements and general guidance for specification of Abstract Test Suites (ATSs), it is useful to
assume a normal form of the process of ATS production leading to a conformance testing specification. This clause describes
the process in just such a normal form. Test suite specifiers are not required to follow this 11orma1 form exactly, however they
are recommended to use a similar process involving the same steps, possibly in a different order.
8.2 For the purposes of this part of ISOIIEC 9646, the ATS production process is assumed to be as follows:
a) study the relevant specifications and ICS proformas to determine what the conformance requirements (including options)
are which need to be tested (see clause 9);
b) decide which test groups will be needed to achieve the appropriate coverage of the conformance requirements (see 10.2);
c) optionally develop test group objectives: the common testing objectives of the elements of each test group (see 10.3);
d) develop test purposes which reflect the test group objectives (if any) of the test groups in which they are contained, and
which provide adequate coverage of the conformance requirements to be tested (see 10.3 and 10.4);
e) choose the abstract testing context and the Abstract Test Method(s) for which the complete abstract test cases need to be
specified, and decide what restrictions need to be placed on the capabilities of the Lower Tester(s) and, if appropriate to the
chosen Abstract Test Methods(s), the Upper Tester(s) and Test Coordination Procedures (see clause 11);
f) apply a standardized test notation to specify the set of abstract test cases, including the test step structure to be used (see
clause 12);
0 ISOAEC
ISO/IEC 9646-2:1994 (E)
g) specify the interrelationships
1) among the test cases,
2) between the test cases and the ICS(s), and
3) as far as possible, between the test cases and the Implementation extra Information for Testing (IXIT) statement(s),
in order to determine the restrictions on the selection and parameterization of test cases for execution, and the restrictions, if
any, 011 the orders in which they can be executed (see clause 14);
h) consider the procedures for maintaining the ATS (see clause 15).
8.3 It is also assumed that during the ATS production process an overall structure for the conformance testing specification(s)
will be developed, with appropriate parts for
a) the Test Suite Structure and Test Purposes (TSS&TP) (see clause 10);
b) one or more ATSs for one or more Abstract Test Methods (see clause 11);
c) the specification of a Test Management Protocol (TMP), if applicable (see clause 13).
8.4 Clauses 9 to 15 provide requirements and guidance which relate to each step in the above process.
8.5 Testing for conformance to a profile is based on the use of appropriate test cases from base specification ATSs for the base
specification(s) referenced in the profile. These base specification ATSs may be either single-protocol or multi-protocol ATSs.
In some cases, it may be appropriate to standardize a base specification ATS for that subset of the base specification used by one
or more specific profile(s), in which case subsequent amendments to the ATS should be made to extend the coverage to meet the
needs of other profiles or the complete base specifications as and when necessary.
Additional profile specific test cases may be needed to address conformance requirements that are relevant to the profile but are
outside the scope of the TSS&TP for any base specifications. These additional profile specific test cases are standardized in the
Profile Specific Test Specification. The requirements and guidance on the production of Profile Test Specifications for protocol
profiles are given in ISO/IEC 9646-6.
9 Conformance requirements and ICS proforma
9.1 Before an ATS can be specified, the test suite specifier shall first determine what the conformance requirements are for the
relevant base specification(s) and what is stated in the ICS proforma(s) concerning the implementation of those specification(s).
9.2 Clauses 6 and 7 specify the requirements to be met by specifiers of the base specifications as a prerequisite to the production
of an ATS for a particular base specification or combination of base specifications.
9.3 If the static conformance requirements are not properly specified, the test suite specifier should contribute to the production
of an amendment to or revision of the relevant specification to clarify the conformance requirements.
10 Test Suite Structure and Test Purposes (TSS&TP)
10.1 Basic requirements
10.1.1 The test suite structure and set of test purposes applicable to all ATSs to be specified for the same base specification or
combination of base specifications shall be specified in the relevant conformance testing specification, preferably in a separate
.
Part
10.1.2 Each ATS shall comprise a number of test cases, each of which is designed to achieve one of the specified test purposes.
The test cases may be grouped into test groups, if necessary nested. The structure shall be hierarchical; thus, an item at a lower
level shall be completely contained within a higher level item. Similar test groups may occur in more than one higher level test
group.
10.1.3 The test suite specifier shall ensure that a subset of the test purposes of each ATS is concerned with capability testing, and
another subset is concerned with behaviour testing. This need not lead to distinct test cases for behaviour and capability testing
because it may be possible to use a combined behaviour and capability test purposes. The test suite specifier shall provide an
explanation of how the test purposes are derived from or relate to the base specification. The test suite specifier shall also provide
a summary of the coverage achieved by the ATS.
10.2 Specification of the test suite structure
10.2.1 In order to ensure that the resulting ATS provides adequate coverage of the relevant conformance requirements, the test
suite specifier is advised to design the test suite structure in terms of nested test groups in a top down manner. There are many
0 ISO/IEC
ISOLIEC 9646-2: 1994 (E)
ways of structuring the same test suite into test groups; no one way is necessarily right and the best approach for one test suite
may not be appropriate for another test suite. Nevertheless, the test suite specifier shall ensure that the test suite includes test
cases for whichever of the following categories are relevant:
a) capability tests (for static conformance requirements);
b) behaviour tests of valid behaviour;
c) behaviour tests that investigate the reaction of the Implementation Under Test (WI’) to invalid test events; these may be
subdivided into those concerned with syntactically invalid test events, semantically invalid test events, and inopportune test
events, as relevant to the protocol concerned;
d) tests focusing on the different roles of the IUT;
e) tests focusing on PDUs sent to the IUT;
f) tests focusing on PDUs received from the IUT;
g) tests focusing on interactions between PDUs sent and PDUs received;
h) tests related to each mandatory capability;
i) tests related to each optional capability;
j) tests related to each protocol phase;
k) variations in the test event occurring in a particular state;
1) timing and timer variations;
m) PDU encoding variations;
n) variations in values of individual parameters;
o) variations in combinations of parameter values;
p) combinations of related requirements from more than one base specification;
q) tests specific for multi-party behaviour.
This list is not exhaustive; additional categories might be needed to ensure adequate coverage of the relevant conformance
requirements for a specific test suite. Furthermore, these categories overlap one another and it is the task of the test suite specifier
to arrange them into an appropriate hierarchical structure.
10.2.2 The following structure is an example of a single-layer protocol test group for a particular role in the Single Party Testing
context, provided for guidance:
A. Capability tests
A. 1 Mandatory capabilities
A.2 Optional capabilities
B. Behaviour tests: response to valid behaviour by peer implementation
B. 1 Connection establishment phase (if relevant)
B.l .l Focus on what is sent to the IUT
B. 1.1.1 Test event variation in each state
B. 1.1.2 Timing/timer variation
B. 1.1.3 Encoding variation
B. 1.1.4 Individual parameter value variation
B. 1.1.5 Combination of parameter values
B.l.2 Focus on what the IUT is requested to send
-
substructured as B. 1.1
B.1.3 Focus on interactions
- substructured as B. 1.1
B.2 Data transfer phase
-
substructured as B. 1
0 ISO/IEC
ISO/IEC 9646-2: 1994 (E)
B.3 Connection release phase (if relevant)
-
substructured as B. 1
C. Behaviour tests: response to syntactically or semantically invalid behaviour by peer implementation
C. 1 Connection establishment phase (if relevant)
C. 1 .l Focus on what is sent to the IUT
C.l. 1 .l Test event variation in each state
C. 1.1.2 Encoding variation of the invalid event
C. 1.1.3 Individual invalid parameter value variation
C. 1.1.4 Invalid parameter value combination variation
C.1.2 Focus on what the IUT is requested to send
C. 1.2.1 Individual invalid parameter values
C. 1.2.2 Invalid combinations of parameter values
C.2 Data transfer phase
- substructured as C.l
C.3 Connection release phase (if relevant)
-
substructured as C. 1
D. Behaviour tests: response to inopportune events by peer implementation
D. 1 Connection establishment phase (if relevant)
D.l .l Focus on what is sent to the IUT
D. 1 .l .l Test event variation in each state
D. 1.1.2 Timing/timer variation
D. 1.1.3 Special encoding variations
D.l .1.4 Major individual parameter value variations
D. 1 .1.5 Variation in major combination of parameter values
D.l.2 Focus on what the IUT is requested to send
- substructured as D. 1.1
D.2 Data transfer phase
-
substructured as D. 1
D.3 Connection release phase (if relevant)
-
substructured as D.l
10.2.3 This test group structure does not cover basic interconnection tests. These may be provided as a list of selected capability
and/or behaviour tests, but shall not involve any additional test purposes.
10.3 Specification of the test purposes
10.3.1 The test suite specifier shall create a set of test purposes, with each test purpose focused on a single conformance
requirement or set of related conformance requirements (e.g. in the case of multi-protocol testing) of the relevant specification(s).
It is suggested that test groups of related test purposes are identified first (as described in 10.2) and that text defining the test group
objective be produced for each test group. Within each test group, several more specific test objectives should be defined, to
become either nested test group objectives or individual test purposes. By successive refinement of the test group objectives in
this way, a structured set of test purposes may be produced.
The test purposes may be produced directly from studying those clauses in the relevant specification(s) which are appropriate to
the test group concerned. For some test groups, the test purposes may be derivable directly from the protocol state table; for
others, they may be derivable from the PDU encoding definitions or the descriptions of particular parameters, or from text which
specifies the relevant conformance requirements.

0 ISO/IEC
ISO/IEC 9646-2: 1994 (E)
This orderly construction technique helps to ensure the adequate coverage of the conformance requirements to be tested. It also
avoids unnecessary duplication of text in the test purposes, because the fuil description of each test purpose does not have to be
written explicitly, but can be assembled by tracing a path down through the nested structure of test group objectives.
NOTE - If the test suite specifier employs a formal description of the base specification(s) concerned, test purposes may be derived from that
by means of some automated method. If an automated method is used, the same requirements apply. However, methods based on Formal
Description Techniques (FDTs) are outside the scope of this part of ISO/IEC 9646. Nevertheless, if an FDT is to be used for this purpose, it
is preferred that it be a standardized one.
10.3.2 In order to increase the efficiency of testing individual parameters on a single PDU, test purposes covering a combination
of parameters may be specified for a single abstract test case. However, the testing of invalid parameter values shall not be
combined with the testing of other valid or invalid values in a single test purpose.
10.3.3 As part of the process of designing the TSS&TP, it is suggested that test purposes be identified initially for each
conformance requirement (e.g. specific parameter) that is to be tested. As a second stage, test purposes for combinations of
related conformance requirements may be specified. If this is done
a) a new test purpose covering a combination of related conformance requirements shall be written referencing those test
purposes which cover the individual conformance requirements;
b) an indication shall be given that one abstract test case is to be produced for that new test purpose, rather than a distinct test
case for each of the referenced test purposes that have been superseded;
c) each superseded test purpose shall remain in the set of test purposes, but shall identify the new test purpose(s) which
supersede it.
10.3.4 The result of identifying and then combining particular conformance requirements to foml test purpose(s) is a
specification of a test suite structure and a list of names of the test cases that shall apply to both the test purposes and to any
ATS produced for those test purposes.
10.3.5 Whatever method is used to derive the test purposes, the test suite specifier should ensure, as far as possible, that they
provide an adequate coverage of the conformance requirements of the relevant specification(s). There shall be at least one test
purpose related to each distinct conformance requirement or set of related conformance requirements.
10.3.6 Test purposes should be specified not only for clearly testable conformance requirements, but also for conformance
requirements that may be untestable using the test methods defined in this part of ISO/IEC 9646.
NOTE - Test purposes for untestable requirements serve to inform the specifiers of the base specifications which conformance requirements
are untestable, by indicating gaps in the ATSs.
10.4 Coverage
Ideally, the TSS&TP should provide coverage of all of the conformance requirements of the relevant base specification(s).
However, if the related ATS(s) are to be developed only to meet the needs of testing particular profile(s), then the TSS&TP may
initially be developed just to provide coverage for those conformance requirements of the base specification(s) relevant to the
particular profile(s). Such a TSS&TP should be expanded to provide full coverage of the base specification(s) when resources
permit.
It is possible to give guidance on the meaning of “adequate” coverage with reference to the test suite structure example in 10.2.
In order to express this, a shorthand notation will be used: the letter “x” will represent all appropriate values for the first digit
in the test group identifier, and similarly “y” for the second digit, so that B.x.y. 1 stands for B. 1.1.1, B. 1.2.1, B.1.3.1, B.2.1 .l,
B.2.2.1, B.2.3.1, B.3.1.1, B.3.2.1 and B.3.3.1.
With this notation, a minimum “adequate” coverage for the example given in 10.2 is considered to be as follows:
a) for capability test groups (A.1, A.2)
1) at least one test purpose per relevant capability,
2) at least one test purpose per relevant PDU type and each major variation of each such type, using “normal” or default
values for each parameter;
b) for test groups concerned with test event variation in each state (B.x.y.1, C.x.1 .l, D.x.y.l), at least one test purpose per
relevant state/event combination;
c) for test groups concemed with timers and timing (B.x.y.2, D.x.y.2), at least one test purpose concerned with the expiry of
each defined timer;
d) for test groups concerned with encoding variations (B.x.y.3, C.x.1.2, D.x.y.3), at least one test purpose for each relevant
kind of encoding variation per relevant PDU type;
0 ISOIIEC
ISO/IEC 9646-2: 1994 (E)
e) for test groups concerned with valid individual parameter values (B.x.y.4, D.x.y.4)
1) for each relevant integer parameter, test purposes concerned with the boundary values and one randomly selected mid-
range value,
2) for each relevant bitwise parameter, test purposes for as many values as practical, but not less than all the “x~ormal” or
common values,
3) for other relevant parameters, at least one test purpose concerned with a value different from what is considered “normal”
or default in other test groups;
NOTE - Tests for valid parameter values should focus on the relevant claims made in the ICS.
f) for test groups concerned with syntactically or semantically invalid individual parameter values (C.x.l.3, C.x.2.1)
1) for each relevant integer parameter, test purposes concerned with invalid values adjacent to the allowed boundary values
defined in the base specification, plus one other randomly selected invalid value,
2) for each relevant bitwise parameter, test purposes for as many invalid values as practical,
3) for all other relevant types of parameter, at least one test purpose per parameter;
NOTE - Tests for invalid parameter values should focus on values outside the range defined in the relevant base specification, rather than
valid values outside the range claimed in the KS.
g) for test groups concerned with combinations of parameter values (B.x.y.5, C.x.1.4, C.x.2.2, D.x.y.5)
1) at least one test purpose for each important combination of specific values (e.g. boundary values),
2) at least one test purpose per set of interrelated parameters to test an arbitrary combination of relevant values.
The test suite specifier shall not assume that the test realizer or test laboratory will perform any checking of test events against
the values specified in the ICS other than that checking which is specified in the abstract test cases. Therefore, the test purposes
and abstract test cases shall make explicit use of values given in the ICS whenever checking of valid parameter values is specified.
The test suite shall include test cases to check for the support of parameter values that are allowed by the base specification(s)
and are within the ranges stated in the ICS. Such test cases shall make use of test suite parameters that contain the relevant ICS
values. The test suite shall also include test cases to check for valid reactions to parameter values th
...