ISO/IEC 9646-1:1991

ISO/IEC
I NTER NATIONAL
STANDARD 96464
First edition
1991 -07- 15
Information technology - Qpen Systems
Interconnection - Conformance testing
methodology and framework --
Part I:
General concepts
Technologies de l'information - Interconnexion de systèmes ouverts -
Cadre général et méthodologie des tests de conformité OS/ -
Partie 1: Concepts généraux
e
Reference number
IS011EC 9646-1:1991 (E)
ISO/IEC 9646-1 : 1991(E)
Contents Page
Foreword . iv
Introduction . v
1 Scope . 1
2 Normative references . 1
3 Definitions . 2
3.1 Reference model definitions . 2
3.2 Terms defined in other International Standards, CClTT
Recommendations and Technical Reports . 2
3.3 Conformance testing definitions . 2
..................................................
3.4 Basic terms 2
3.5 Typesoftesting . 3
3.6 Terminology of test suites . 3
3.7 Terminology of results .
3.8 Terminologyof test methods . 5
4 Abbreviations . 6
5 The meaning of conformance in OS1 . 6
5.1 Introduction . 6
5.2 Conformance requirements . 6
5.3 Static conformance requirements . 6
5.4 Dynamic conformance requirements . 7
5.5 Protocol Implementation Conformance Statement (PICS) . 7
5.6 A Conforming system . 7
5.7 Interworking and conformance . 7
6 Conformance and testing . 7
6.1 Objectives of conformance testing . 7
6.1.1 Introduction . 7
6.1.2 Basic Interconnection Tests (BITS) . 8
6.1.3 Capability tests . 8
6.1.4 Behaviour tests . 8
6.1.5 Conformance resolution tests . 8
6.2 Protoool Implementation Extra Information for Testing (PIXIT) . 9
6.3 Conformance assessment process overview . 9
6.4 Use of BITS and capability tests in the test campaign . 9
6.5 Analysis of results . 11
6.5.1 Test outcomes and test verdicts . 11
6.5.2 Conformance test reports . 11
0 ISO/IEC 1991
All rights reserved . Unless otherwise specified. no part of this publication may be reproduced or
in any form or by any means. electronic or mechanical. including photocopying and
utilized
microfilm. without permission in writing from the publisher .
ISO/IEC Copyright Office 0 Case postale 56 e CH-1211 Genève 20 0 Switzerland
Printed in Switzerland
li
ISO/IEC 9646-1 : 1991 (E)
6.5.3 Repeatability of results . 11
6.5.4 Comparabiïiy of results . 11
6.5.5 Auditability of results . 11
7 Testmethods . 11
7.1 Introduction . 11
7.2 Classification of real open systems and IUTs for conformance testing . 13
7.2.1 Classification of SUTs . 13
7.2.2 Identification of the IUT . 13
7.3 Abstract testing methodology . 13
7.3.1 General . 13
7.3.2 Points of Control and Observation (PCOs) . 13
7.4 Abstract testing functions . 15
7.5 Overview of Abstract Test Methods . 15
7.5.1 End-system IUTs . 15
7.5.2 The Local and Distributed test methods . 15
7.5.3 The Coordinated and Remote test methods . 15
7.5.4 Variants of end-system test methods . 15
7.5.5 Relay-system IUTs . 15
7.6 Applicability of test methods to real open systems . 15
7.7 Applicability of the test methods to OS1 protocols and layers . 17
8 Testsuites . 17
8.1 Structure . 17
8.2 Abstract and executable test cases . 17
9 Relationships between parts. concepts and roles . 17
10Compliance . 19
Annexes
A (informative) Options . 21
B (informative) Applicability of the test methods to OS1 protocols . 22
B.l The Physical layer . 22
6.2 Data Link and Media Access Control protocols . 22
6.3 Network protocols . 22
6.4 Transport protocol . 22
B.5 Session protocol . 22
6.6 Presentation and Application protocols . 22
6.6.1 General comments . 22
8.6.2 Presentation . -23
8.6.3Application. . 23
6.6.4 Transfer syntaxes . 23
B.7 Connectionless protocols . 23
C (informative) Index to ISOIIEC 9646 Parts 1.2. 4 and 5 . 24
C.1 Introduction . 24
C.2Thelndex . 24
ill
ISO/IEC 9646-1 : I991 (E)
Foreword
IS0 (the International Organization for Standardization) and IEC (ihe
International Electrotechnical Commission) form the specialized system
for worldwide standardization. National bodies that are members of IS0
or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal
with particular fields of technical activity. IS0 and IEC technical com-
mittees collaborate in fields of mutual interest. Other international or-
ganizations, governmental and non-governmental, in liaison with IS0
and IEC, also take part in the work.
In the field of information technology, ISQ and IEC have established a
joint technical committee, ISO/IEC JTC I. Draft International Standards
adopted by the joint technical cornmittee are circulated to national bod-
ies for voting. Publication as an International Standard requires ap-
proval by at least 75 % of the national bodies casting a vote.
International Standard ISO/IEC 9646-1 was prepared by Joint Technical
Committee ISO/IEC JTC 1, Information technology.
ISO/IEC 9646 consists of the following parts, under the general title In-
formation technology - Open Systems Inferconnection - Conformance
testing methodology and framework:
- Part 1: General concepts
- Part 2: Abstract test suite specificailon
- Part 3: The Tree and Tabular Combined Notation (TTCN)
- Part 4: Test realization
- Part 5: Requirements on test laboratories and clients for the con-
formance assessment process
Annexes A, B and C of this pari of ISO/IEC 9646 are for infarmation only.
iv
ISO/IEC 9646-1 : 1991 (E)
Introduction
The objective of OS1 will not be completely achieved until systems can be tested to
determine whether they conform to the relevant protocol specification(s). The rele-
OS1 International Standards or CCITT Recommendations.
vant ones can be
Standardized abstract test suites should be developed for each International Stand-
ard or CCITT Recommendation which specifies an OS1 protocol, for use by suppliers
OS1 products, by telecommunications
or implementors in self-testing, by users of
administrations and recognized private operating agencies, or by other third party
testing organizations. This should lead to comparability and wide acceptance of test
results produced by different test laboratories, and thereby minimize the need for
repeated conformance testing of the same system.
The standardization of test suites requires international definition and acceptance of
a common testing methodology, together with appropriate testing methods and pro-
cedures. It is the purpose of ISOAEC 9646 to define the methodology, to provide a
framework for specifying conformance test suites, and to define the procedures to
be followed during testing.
Conformance testing involves testing both the capabilities and behaviour of an
implementation, and checking what is observed against both the conformance
requirements in the relevant International Standards or CCllT Recommendations
and what the implementor states the implementation’s capabilities are.
Conformance testing does not include assessment of the performance nor the
robustness or reliability of an implementation. It cannot give judgements on the phys-
ical realization of the abstract service primitives, how a system is implemented, how
it provides any requested service, nor the environment of the protocol implementa-
tion. It cannot, except in an indirect way, prove anything about the logical design of
the protocol itself.
The purpose of conformance testing is to increase the probability that different OS1
implementations are able to interwork. However it should be borne in mind that the
complexity of most protocols makes exhaustive testing impractical on both technical
and economic grounds. Also, testing cannot guarantee conformance to a specifica-
tion since it detects errors rather than their absence. Thus conformance to a test
suite alone cannot guarantee interworking. What it does do is give confidence that
an implementation has the required capabilities and that its behaviour conforms con-
sistently in representative instances of communication.
It should be noted that the OS1 basic reference model (IS0 7498: 1984 or CClTT
X.200(1984)) states (in 4.3):
“Only the external behaviour of Open Systems is retained as the
standard of behaviour of real Open Systems”
This means that although aspects of both internal and external behaviour are
described in OS1 International Standards and CClTT Recommendations, it is only
the requirements on external behaviour that have to be met by real open systems.
Although some of the methods defined in ISO/IEC 9646-2 do impose certain limita-
tions on the implementor, for example that there be some means of realizing control
V
ISO/IEC 9646-1 : 1991 (E)
and observation at one or more service access points, it should be noted that other
methods defined herein do not impose such limitations.
However, in the case of partial OS1 end-systems which provide OS1 protocols up to
a specific layer boundary, it is desirable to test not only the external behaviour of the
implemented protocol entities, but also the ability of those entities to support correct
external behaviour in higher layers.
Detailed investigation of relative benefits, efficiency and limitations of all methods is
addressed in various parts of ISO/IEC 9646. However, any organization contemplat-
ing the use of test methods defined in ISOIIEC 9646-2 in a context such as certifica-
tion should carefully consider the limitations on their applicability and the benefits of
each.
Testing is voluntary as far as ISO/IEC and CClTT are concerned. Requirements for
testing in procurement and other external contracts are not a matter for standardiza-
tion.
This pari of ISO/IEC 9646 is also to be published by CClTT as Recommendation
X.290 (1991).
vi
INTERNATIONAL STANDARD
ISO/IEC 9646-1 : 199I(E)
~~ ~~ ~
Information technology - Qpen Systems Interconnection -
Conformance testing methodology and framework -
Part I:
General concepts
b) to achieve comparability between the results of the cor-
1 Scope
responding tests on a paiticular OS1 implementation
1.1 ISO/IEC 9646 specifies a general methodology for test-
applied in different places at different times;
ing the conformance of products to International Standards or
c) to facilitate communication between the parties respon-
CCITT Recommendations that specify OS1 protocols which
1.3 above for parts 2 to
sible for the activities described in
the products are claimed to implement. The methodology
5.
also applies to testing conformance to an International Stand-
ard or CCITT Recommendation that specifies a transfer syn-
1.6 This part of ISO/IEC 9646 includes tutorial introductory
tax to the extent that can be determined by testing it in com-
material which provides
bination with a specific OS1 protocol.
a) an exposition of the meaning of conformance in the
1.2 The contents of ISO/IEC 9646 are also, in principle, appli-
context of OSI;
cable to conformance testing for ISDN two-party protocols.
b) a description of the major categories of conformance
1.3 ISO/IEC 9646 is applicable to the different phases of the
tests;
conformance testing process, these phases being character-
c) an introduction to the conformance assessment proc-
ized by three major activities. These activities are
ess;
a) the specification of abstract test suites for particular
d) an introduction to the Abstract Test Methods and their
OS1 protocols;
applicability;
b) realization of the means of executing specific test
e) an introduction to the concepts of test suite design.
suites;
In addition, this part describes the relationship between the
c) the conformance assessment process carried out by a
other parts of ISO/IEC 9646 and the activities involved in
test laboratory for a specific client, culminating in the pro-
conformance testing, and introduces the concept of compli-
duction of a Protocol Conformance Test Report, which
ance with respect to the other parts of ISO/IEC 9646.
gives the results in terms of the protocol specification and
1.7 The following are outside the scope of ISO/IEC 9646:
test suite used.
a) certification, an administrative procedure which may
ISOllEC 9646 is structured into five separate parts, each of
follow conformance testing;
which, apart from part 1, is applicable to just one of these
activities.
b) requirements for procurement and contracts;
This part of ISO/IEC 9646 is applicable to all three activities,
c) testing by means of test methods which are specific to
providing tutorial introductory material, together with defini-
particular applications, protocols or systems;
tions of common terms and concepts.
d) testing of non-protocol conformance requirements;
NOTE - ISO/IEC 9646-2 deals with the requirements and guidance
e) test methods that involve more than two endsystems
for the specification of abstract test suites, independent of test nota-
communicating together.
tion. ISO/IEC 96483 defines the recommended test notation. ISO/
IEC 96464 deals with requirements and guidance for realization of
NOTE - ISOllEC 9646 does not apply fully to Physical layer proto-
the Means of Testing, and ISO/IEC 96465 deals with requirements
cols. Nevertheless, many of the concepts apply to all protocols.
and guidance for test laboratories and their clients for the conform-
ance assessment process.
2 Normative references
1.4 ISO/IEC 9646 specifies the requirements for and gives
guidance on the procedures to be followed in OS1 conform- The following standards contain provisions which, through
ance testing. reference in this text, constitute provisions of this part of ISO/
IEC 9646. At the time of publication, the editions indicated
1.5 ISO/IEC 9646 includes only such information as is nec-
were valid. All standards are subject to revision, and parties
essary to meet the following objectives:
to agreements based on this part of ISO/IEC 9646 are
a) to achieve an adequate level of confidence in the tests
encouraged to investigate the possibility of applying the most
as a guide to conformance;
recent editions of the standards listed below. Members of
IEC and IS0 maintain registers of currently valid International
Standards.
iSO/IEC 9646-1 : 199l(E)
s) Transport layer
IS0 7498:1984, Information processing systems -Open Sys-
tems Interconnection - Basic Reference Model.
3.2 Terms defined in other International
(See also CClTT Recommendation X.200)
Standards, CCllT Recommendations and
ISO/TR 8509:1987, Information processing systems - Open
Technical Reports
Systems Interconnection - Service Conventions.
(See also CClTT Recommendation X.210) This part of ISOAEC 9646 uses the following terms defined in
the OS1 Service Conventions (ISOTTR 8509 / CClTT X.21 O):
IS0 88251990, Information technology - Open Systems
Interconnection - Specification of Basic Encoding Rules for a) service-user
Abstract Syntax Notation One (ASN. 1).
b) service-provider
(See also CClll Recommendation X.209)
c) service primitive
ISOIIEC 9646-2:1991 , Information technology - Open Sys-
This part of ISWIEC 9646 uses the following term defined in
tems Interconnection - Conformance testing methodology
the ASN.l - Basic Encoding Rules (IS0 8825 / CClTT
and framework - fart 2: Abstract Test Suite specification.
X.209):
(See also CClll Recommendation X.291(1991))
- encoding
ISO/I EC 9646-3:-', Information technology - Open Systems
Interconnection - Conformance testing methodology and
3.3 Conformance testing definitions
framework - Part 3: The tree and tabular combined notation.
For the purposes of this part of ISO/IEC 9646 the definitions
I SOI1 EC 9646-4: 1 991 , Information technology - Open Sys-
in 3.4 to 3.8 apply.
tems Interconnection - Conformance testing methodology
and framework - Part 4: Test Realization.
3.4 Basic terms
(See also CClTT Recommendation X.293(1991))
3.4.1 implementation under test [IUT] : An implementation
ISO/IEC 9646421991 , Information technology - Open Sys-
of one or more OS1 protocols in an adjacent user/provider
tems Interconnection - Conformance testing methodology
relationship, being that part of a real open system which is to
and framework - Part 5: Requirements on test laboratories
be studied by testing.
and clients for the Conformance Assessment Process.
3.4.2 system under test [SUI : The real open system in
(See also CCllT Recommendation X.294(1991))
which the IUT resides.
3.4.3 dynamic conformance requirement : One of the
3 Definitions
requirements which specifies what observable behaviour is
permitted by the relevant OS1 International Standard(s) or
3.1 Reference model definitions
CClTT Recommendation(s) in instances of communication.
This part of ISO/IEC 9646 is based upon the concepts devel-
3.4.4 static conformance requirement : One of the require-
oped in the Basic Reference Model for Open Systems Inter-
ments that specify the limitations on the combinations of
connection (IS0 7498 / CClTT X.200), and makes use of the
implemented capabilities permitted in a real open system
following terms defined in that international Standard:
which is claimed to conform to that OS1 International Stand-
a) (N)-entity
ard or CClTT Recommendation.
b) (N)-layer
3.4.5 capability (of an Implementation) : A set of functions
e
in the relevant protocol(s) which is supported by the imple
c) (N)-protocol
mentation.
d) (N)-protocoldata-unit
3.4.6 protocol implementation conformance statement
e) (N)-relay
[PICS] : A statement made by the supplier of an OS1 imple-
9 (N)-service
mentation or system, stating which capabilities have been
implemented, for a given OS1 protocol.
g) (N)-service-access-point
3.4.7 PICS proforma : A document, in the form of a question-
h) Application layer
naire, designed by the protocol specifier or conformance test
i) Application-service-element
suite specifier, which when completed for an OS1 implemen-
tation or system becomes the PICS.
j) Data Link layer
k) Network layer 3.4.8 protocol implementation extra information for test-
ing [PIXIT] : A statement made by a supplier or implementor
I) Physical layer
of an IUT which contains or references all of the information
m) Presentation layer
(in addition to that given in the PICS) related to the IUT and
its testing environment, which will enable the test laboratory
n) real open system
to run an appropriate test suite against the IUT.
O) real system
3.4.9 PlXlT proforma : A document, in the form of a ques-
p) Session layer
tionnaire, provided by the test laboratory, which when com-
q) subnetwork
pleted during the preparation for testing becomes a PIXIT.
r) Transfer syntax
1) To be published.
ISO/IEC 9646-1 : I991 (E)
3.4.10 conforming implementation : An IUT which satisfies
3.6 Terminology of test suites
both static and dynamic conformance requirements, consist-
3.6.1 (abstract) test method [ATUI] : The description of how
ent with the capabilities stated in the PICS.
an IUT is to be tested, given at an appropriate level of
3.4.11 system conformance statement [SCS] : A docu-
abstraction to make the description independent of any par-
ment summarizing which OS1 International Standards or
ticular realization of a Means of Testing, but with enough
CClTT Recommendations are implemented and to which
detail to enable tests to be specified for this test method.
ones conformance is claimed.
3.6.2 abstract testing methodology : An approach to
3.4.12 client (of a test laboratory) : The organization that
describing and categorizing Abstract Test Methods.
submits a system or implementation for conformance testing.
3.6.3 abstract test case : A complete and independent
3.4.13 test laboratory : An organization that carries out con-
specification of the actions required to achieve a specific test
formance testing. This can be a third party, a user organiza-
purpose (or a specified combination of test purposes),
tion, a telecommunications administration or recognised pri-
defined at the level of abstraction of a particular Abstract Test
vate operating agency, or an identifiable part of a supplier
Method, starting in a stable testing state and ending in a sta-
organization.
ble testing state. This specification may involve one or more
consecutive or concurrent connections.
3.5 Types of testing
NOTES
3.5.1 static conformance review : A review of the extent to
1 The specification should be complete in the sense that ii is suffi-
which the static conformance requirements are met by the
cient to enable a test verdict to be assigned unambiguously to each
IUT, accomplished by comparing the PICS with the static
potentially observable test outcome (i.e. sequence of test events).
conformance requirements expressed in the relevant Interna-
2 The specification should be independent in the sense that ii should
tional Standard(s) or CClTT Recommendation(s).
be possible to execute the derived executable test case in isolation
3.5.2 basic interconnection test [BIT] : A test of an IUT from other such test cases (i.e. the specificatin should always in-
clude the possibility of starting and finishing in the "idle" state).
which has limited scope to determine whether or not there is
sufficient conformance to the relevant protocol(s) for inter-
3.6.4 executable test case : A realization of an abstract test
connection to be possible, without trying to perform thorough
case.
testing.
NOTE - In general the use of the word "test" in ISOAEC 9646 will
3.5.3 capability test : A test to verify the existence of one or
imply its normal English meaning. Sometimes it may be used as an
more claimed capabilities of an IUT.
abbreviation for abstract test case or executable test case. The con-
text should make the meaning clear.
NOTE - capability testing involves checking all mandatory capabili-
ties and those optional ones that are stated in the PICS as supported,
3.6.5 test purpose : A prose description of a narrowly
but not checking those optional ones which are stated in the PICS as
defined objective of testing, focusing on a single conform-
not supported by the IUT.
ance requirement as specified in the appropriate OS1 Interna-
tional Standard or CClTT Recommendation (e.g. verifying
3.5.4 behaviour test : A test to determine the extent to which
the support of a specific value of a specific parameter).
one or more dynamic conformance requirements are met by
the IUT.
3.6.6 test group objective : A prose description of the com-
mon objective which the test purposes within a specific test
3.5.5 conformance resolution test : A non-standardized,
group are designed to achieve.
possibly systemspecific test to fulfil a test purpose for which
a standardized abstract test case is not defined, in order to
3.6.7 generic test case : A specification of the actions
investigate the behaviour of an OS1 protocol implementation
required to achieve a specific test purpose, defined by a test
with respect to one or more particular conformance require-
body together with a description of the initial testing state in
ments.
which the test body is to start.
3.5.6 conformance testing : Testing the extent to which an
3.6.8 (test) preamble : The sequences of test steps from the
IUT is a conforming implementation.
starting stable testing state of the test case up to the initial
testing state from which the test body will start.
3.5.7 conformance assessment process : The complete
process of accomplishing ail conformance testing activities
3.6.9 test body : The sequences of test steps that achieve
necessary to enable the conformance of an implementation
the test purpose.
or a system to one or more OS1 International Standards or
3.6.10 (test) postambie : The sequences of test steps from
CClTT Recommendations to be assessed.
the end of the test body up to the finishing stable testing
3.5.8 test campalgn : The process of executing the Param-
state(s) for the test case.
eterized Executable Test Suite for a particular IUT and pro-
3.6.11 test step : A named subdivision of a test case, con-
ducing the conformance log.
structed from test events and/or other test steps.
3.5.9 embedded testing : Testing specified for a single-pro-
3.6.12 test event : An indivisible unit of test specification at
tocol within a multi-protocol IUT including the specification of
the level of abstraction of the specification (e.g. sending or
the protocol activity above the one being tested, but without
receiving a single PDU).
specifying control or observation at service boundaries within
the multi-protocol IUT. 3.6.13 unidentified test event : A test event which is used
to provide for receipt of PDUs and/or ASPS without identify-
NOTE -This definition assumes that the protocols of the IUT are or-
ing them in the test case.
dered in a continuous adjacent user/provider relationship.
ISO/IEC 9646-1 : 1991 (E)
NOTE - In TTCN, the unidentified test event is the Otherwise state- 3.6.30 parameterized executable test suite [PETS] : A
ment.
SETS, in which all test cases have been parameterized in
accordance with the appropriate PICS and PIXIT, and corre-
3.6.14 testing state : A state encountered during testing,
sponding to a PATS.
comprising the combination of the states of the SUT, the test
system, the protocols for which control and observation is
3.6.31 standardized abstract test suite [ATS] : An abstract
specified in the ATS, and, if relevant, the state of the underly-
test suite specified within an International Standard or CClTT
ing service.
Recommendation or, in the absence of such an International
Standard or CClTT Recommendation, within a publicly avail-
3.6.15 stable testing state : A testing state which can be
able document which is in the process of being standardized
maintained, without prescribed Lower Tester behaviour, suf-
within ISOAEC or CCITT, and which has the highest stand-
ficiently long to span the gap between one test case and the
ardization status currently available, having reached at least
next in a test campaign.
the committee drafî, draft proposal or draft recommendation
3.6.16 idle testing state : A stable testing state in which there
status.
is no established connection of the reievant protocol(s) and in
3.6.32 conformance testing standard : The International
which the state of the SUT is independent of any previously
Standard or CClTT Recommendation or draft thereof that
executed test cases.
contains a standardized ATS.
3.6.17 transient testing state : Any testing state which is not
a stable testing state.
3.7 Terminology of results
NOTE - Transient testing states include those testing states that are
3.7.1 repeatability (of results) : Characteristic of a test
(b
in the middle of a logical exchange of PDUs (e.g. to establish a con-
case, such that repeated executions on the same IUT under
nection or perform negotiation), particularly when a request PDU has
the same conditions lead to the same test verdict, and by
been sent (or received) and the corresponding response PDU has
extension a characteristic of a test suite.
not bean received (or sent).
3.7.2 comparability (of results) : Characteristic of conform-
3.6.18 initiai testing state : The testing state in which a test
ance assessment processes, such that their execution on the
body starts.
same IUT, in different test environments, leads to the same
NOTE -This may be either a stable testing state or a transient state.
overall summary of conformance for the specified IUT.
3.6.19 (conformance) test suite : The complete set of test
3.7.3 (observed) test outcome : The sequence of test
cases, possibly combined into nested test groups, that is
events, together with associated data and/or parameter val-
needed to perform dynamic conformance testing for one or
ues, which occurred during test execution of a specific
more OS1 protocols.
parameterized executable test case.
NOTE - It should cover both capability testing and behaviour testing.
3.7.4 foreseen test outcome : An observed test outcome
It may be qualified by the adjeciiies: abstract, generic or executable,
identified in the abstract test case.
as appropriate. Unless stated otherwise, an “abstract test suite” is
meant. NOTE - A foreseen test outcome may include an unidentified test
event.
3.6.20 test case : A generic, abstract or executable test case.
3.7.5 unforeseen test outcome : An observed test outcome
3.6.21 test group : A named set of related test cases.
not identified in the abstract test case.
3.6.22 generic test suite : A test suite composed of generic
NOTE -An unforeseen test outcome can only lead to a test case er-
test cases.
e
ror or an abnormal test case termination.
3.623 abstract test suite [ATS] : A test suite composed of
3.7.6 (test) verdict : A statement of ”pass”, “fail” or “inconclu-
abstract test cases.
sive”, specified in an abstract test case, concerning conform-
3.6.24 executable test suite [ETS] : A test suite composed ance of an IUT with respect to that test case when it is exe-
of executable test cases. cuted.
3.6.25 selected abstract test suite [SATS] : The subset of 3.7.7 system conformance test report [SCTR] : A docu-
an ATS selected using a specific PICS and PIXIT. ment, written at the end of the conformance assessment
process, giving an overall summary of the conformance of
3.6.26 selected executable test suite [SETS] : The subset
the system or implementation to the set of protocols for which
of an ETS selected using a specific PICS and PIXIT.
conformance testing was carried out.
3.6.27 parameterized abstract test case : An abstract test
3.7.8 protocol conformance test report [PCTR] : A docu-
case in which all appropriate parameters have been supplied
ment, written at the end of the conformance assessment
with values in accordance with a specific PICS and PIXIT.
process, giving the details of the testing carried out for a par-
3.6.28 parameterized executable test case : An executable
ticular protocol. It lists all of the abstract test cases and iden-
test case, in which all appropriate parameters have been sup-
tifies those for which corresponding executable test cases
plied with values in accordance with a specific PICS and
were run, together with the verdicts assigned to each test
PIXIT, and corresponding to a parameterized abstract test
case executed.
case.
3.7.9 valid test event : A test event which is allowed by the
3.6.29 parameterized abstract test suite [PATS] : A SATS
protocol specification, being both syntactically and semanti-
in which all test cases have been parameterized in accord-
cally correct, and occurring when allowed to do so by the pro-
ance with the appropriate PICS and PIXIT.
tocol specification.
ISOAEC 96461 : 199l(E)
NOTE -The underlying service-provider is immediately beneath the
3.7.10 invalid test event : A test event that violates at least
(lowest layer) protocoi which is the focus of testing. It may use one
one conformance requirement of the relevant protocol or
or more OS1 layers, or only the Physical medium.
transfer syntax specification.
3.8.3 upper tester [UT] : The representation In IÇO/IEC
NOTE -This term is not to be confused with the term “invalid event”
9646 of the means of providing, during test execution, con-
as defined in IS0 7776.
trol and observation of the upper service boundary of the IUT,
3.7.11 inopportune test event : A test event which occurs
as defined by the chosen Abstract Test Method.
to do so by the protocol specification.
when not allowed
3.8.4 abstract (N)-servlce-primitive [(N)-ASP] : An imple-
3.7.12 syntactically Invalid test event : A test event which
mentation-independent description of an interaction between
syntactiiily is not allowed by the protocol specification.
a service-user and a servke-provider at an (N)-service
3.7.13 semantically invalid test event : A test event which
boundary, as defined in an OS1 service definition.
is neither inopportune nor syntactically invalid, but which con-
3.8.5 test coordination procedures : The rules for cooper-
tains a semantic error with respect to the relevant protocol
ation between the Lower and Upper Testers during testing.
specification (e.g. a PDU containing a parameter value out-
3.8.6 test management protocol Wpl : A protocol which
side the negotiated range for that parameter).
is used in the test coordination procedures for a particular
3.7.14 pass (verdict) : A test verdict given when the
test suite.
observed test outcome gives evidence of conformance to the
3.8.7 test system : The real system which includes the real-
conformance requirement(s) on which the test purpose(s) of
ization of the Lower Tester.
the test case is (are) focused, and when all the test events are
.1)
valid with respect to the relevant International Standard(s) or
NOTE -The same test system can be used as part of several Means
CÇITT Recommendation(s). of Testing.
3.7.15 fail (verdict) : A test verdict given when the observed 3.8.8 local test method : An Abstract Test Method in which
test outcome either demonstrates nonconformance with
both the Lower and Upper Testers are located within the test
to (at least one of) the conformance requirement(s)
respect system and there is a PCO at the upper service boundary of
on which the test purpose(s) of the test case is (are) focused, the IUT.
or contains at least one invalid test event, with respect to the
3.8.9 distributed test method : An Abstract Test Method in
relevant International Standard(s) or CCllT Recommenda-
which the Upper Tester is within the SUT and there is a PCO
tion(s).
at the upper service boundary of the IUT.
3.7.16 inconclusive (verdict) : A test verdict given when the
3.8.10 coordlnated test method : An Abstract Test Method
observed test outcome is such that neither a pass nor a fail
in which the Upper Tester is within the SUT and for which a
verdict can be given.
standardized TMP is defined for the test coordination proce-
3.7.17 test case error : The term used to describe the result dures, enabling the control and observation to be specified
of execution of a test case when an error is detected in the solely in terms of the Lower Tester activity, including the con-
test case itself. trol and observation of Test Management PDUs.
3.7.18 abstract test case error : A test case error resulting
3.8.11 remote test method : An Abstract Test Method in
from an error in the abstract test case. which the control and observation of test events is specified
solely in terms of Lower Tester activity, and in which some
3.7.19 executable test case error : A test case error in the
requirements for test coordination procedures may be
O
realization of an abstract test case.
implied or informally expressed in the ATS, but in which no
3.7.20 abnormal (test case) termination :The term used to
assumption is made regarding their feasibility or realization.
describe the result of execution of an abstract test case when
3.8.12 means of testing [NIOTJ (IUTs) : The combination of
it has been prematurely terminated by the test system.
equipment and procedures that can perform the derivation,
3.7.21 conformance log : A human-readable record of infor-
selection, parameterization and execution of test cases, in
mation produced as a result of a test campaign, which is suf-
conformance with a reference standardized ATS, and can
ficient to record the observed test outcomes and verify the
produce a conformance log.
assignment of test results (including test verdicts).
3.8.13 test realization : The process of producing a Means
of Testing IUTs.
3.8 Terminology of test methods
3.8.14 reference standardized (OW) abstract test suite
3.8.1 point of control and observation [PCO] : A point
within a testing environment where the occurrence of test [ATS] : The standardized ATS for which a Means of Testing
is realized.
events is to be controlled and observed, as defined in an
Abstract Test Method.
3.8.15 test realizer : An organization which takes responsi-
NOTE -A PCO is characterized by the set of ASPS andior PDUs ihai bility for providing, in a form independent of the clients of a
can occur, according to the ATS, at that PCO.
test laboratory and their IUTs, a Means of Testing IUTs in
conformance with an ATS.
3.8.2 lower tester [L‘il : The representation in ISOllEC 9646
of the means of providing, during test execution, indirect con-
3.8.16 comprehensive testing service : A service, offered
trol and observation of the lower service boundary of the IUT
to clients by a test laboratory, to perform the conformance
via the underlying service-provider.
assessment process for one or more OS1 protocoi(s), with a
choice of test methods sufficient to make the service applica-
ISO/IEC 9646-1 : 1991 (E)
ble to all real open systems that claim to implement the spec- OS1 International Standards and CCITT Recommendations
ified protocols. form a set of inter-related International Standards and CClTT
Recommendations which together define behaviour of open
systems in their communication. Conformance of a real sys-
4 Abbreviations
tem is, therefore, expressed at two levels, conformance to
each individual International Standard or CClTT Recommen-
dation, and conformance to the set.
NOTE - If the implementation is based on a predefined set of Inter-
national Standards or CCllT Recommendations, often referred to as
a functiinal standard or profile, the concept of conformance can be
extended to specific requirements expressed in the hinctiinal stand-
ard or profile, as long as they do nut confli with the requirements of
ASP : abstract-service-primitive
the base (protocol) International Standards and CCllT Recommen-
daüons.
ATM : abstract test method
ATS : abstract test suite
5.2 Conformance requirements
BIT : basic interconnection test
5.2.1 The conformance requirements in an International
Standard or CCITT Recommendation can be
DTE : data terminal equipment
ETS : executable test suite a) mandatory requirements: these are to be observed in
all cases;
ISDN : integrated services digital network
b) conditional requirements: these are to be observed if
IUT : implementation under test
the conditions set out in the specification apply;
I
LT : lower tester
c) options: these can be selected to suit the implementa-
MOT : means of testing
to the
tion, provided that any requirements applicable
option are observed. More information on options is pro-
OS1 : open systems interconnection
vided in Annex A.
PATS : parameterized abstract test suite
For example, CClTT essential facilities are mandatoiy
PCO : point of control and observation
requirements; additional facilities can be either conditional or
PCTR : protocol conformance test report
optional requirements.
PDU : protocol data unit
NOTE - The CCl7-i terms “essential facilities” and “additional facili-
ties” need to be considered in the context of the scope of the CClll
PETS : parameterized executable test suite
Recommendation concerned; for example, in many cases, essential
PICS : protocol implementation conformance statement
facilities are mandatory for networks but not for DTEs.
PlXlT : protocol implementation extra information for testing 5.2.2 Furthermore, conformance requirements in an Interna-
tional Standard or CClTT Recommendation can be stated
SAP : service-access-point
a) positively: they state what shall be done;
SATS : selected abstract test suite
b) negatively: they state what shall not be done.
SCS : system conformance statement
5.2.3 Finally, conformance requirements fall into two groups:
SCTR : system conformance test report
a) static conformance requirements;
SETS : selected executable test suite
b) dynamic conformance requirements.
SUT : system under test
These are discussed in 5.3 and 5.4, respectively.
ThlP : test management protocol
TWDU : test management PDU
5.3 Static conformance requirements
TiCN : tree and tabular combined notation
Static conformance requirements are those that specify the
limitations on the combinations of implemented capabilities
UT : upper tester
permitted in a real system which is claimed to conform to the
or CClTT Recommenda-
relevant OS1 International Standard
5 The meaning of conformance in OS1
tion. They define the allowed minimum capabilities in order to
facilitate interworking. They may be specified at a broad
5.1 Introduction
level, such as the grouping of PDUs into functional units or
In the context of OSI, a real system is said to exhibit conform- protoail classes, or at a detailed level, such as a range of val-
ues that have to be supported for specific parameters or tim-
ance if it complies with the requirements of applicable OS1
International Standards or CClTT Recommendations in its ers.
communication with other real systems.
Static conformance requirements in OS1 International Stand-
Applicable OS1 International Standards or CClTT Recom- ards or CClTT Recommendations can be of two varieties:
mendations include those that specify OS1 protocols, and
a) those which determine the capabilities to be included in
those that specify OS1 transfer syntaxes (insofar as they are
the implementation of the particular protocol;
implemented in conjunction with protocols).
ISO/IEC 96461 : 199l(E)
b) those which determine multi-layer dependencies, e.g.
5.7 Interworking and conformance
those which place limitations on the capabilities of the
5.7.1 The
...