ISO/TR 12310:2015

TECHNICAL ISO/TR
REPORT 12310
First edition
2015-05-15
Health informatics — Principles and
guidelines for the measurement of
conformance in the implementation of
terminological systems
Informatique de santé — Principes et lignes directrices pour le
mesurage de la conformité dans la mise en oeuvre des systèmes
terminologiques
Reference number
©
ISO 2015
© ISO 2015, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO 2015 – All rights reserved

Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Objective . 1
3 Terms and definitions . 1
4 Purposes for conformance . 3
4.1 Interoperability . 4
4.2 Data analysis . 4
4.3 Consistency of user experience . 4
4.4 Application functionality . 4
4.5 Acceptance filter . 4
5 Conformance process . 4
5.1 Documenting expectations . 4
5.1.1 Optionality . 5
6 Terminology artefact conformance considerations . 6
6.1 Code system considerations . 6
6.1.1 What is the code system being referenced? . 6
6.1.2 What version(s) of the code system are supported? . 6
6.1.3 How are codes represented? . 7
6.1.4 What are the version migration expectations, if any? . 7
6.1.5 What is the authoritative source that will be used for processing updates
to a given terminology? . 7
6.1.6 What status must usable code system elements have? . 8
6.1.7 Which codes and concepts are usable for what purpose? . 8
6.1.8 What representations are allowed for what purposes? . 8
6.1.9 What code system relationships must be understood and navigated?. 8
6.1.10 Are the semantics clearly defines? . 9
6.1.11 What are the expectations for post-coordination? . 9
6.1.12 What partitions are included? . 9
6.1.13 What code system supplements are supported? .10
6.2 Reference sets: sets of codes that are allowed .10
6.2.1 What code systems are drawn from? .10
6.2.2 How is the reference set defined? .10
6.2.3 Which representations are allowed from a code system?.11
6.2.4 What constraints are there on post-coordinated concepts? .11
6.2.5 What is post-coordinated vs. Pre-coordinated? .11
6.2.6 What happens if the concept exists more than once in the reference set? .11
6.3 Reference set bindings .11
6.3.1 Is the binding to the reference set static or dynamic? .12
6.3.2 What reference set representation capabilities are supported? .12
6.3.3 Is the reference set extensible? .12
6.3.4 What expectation is there to support all codes within the bound reference set? 12
6.4 What is the reference set bound to? .13
6.5 When and where does the binding apply? .13
7 Terminology usage conformance considerations .13
7.1 Data capture .14
7.1.1 Are the code system contents expected to be exposed directly?.14
7.1.2 What aspects may be or must be exposed to users? .14
7.1.3 Are the available codes to be displayed in a particular order? .14
7.1.4 Are there constraints on how the codes are to be navigated? .14
7.1.5 Are deprecated, retired, or pending codes expected to be presented
differently? .14
7.1.6 Shall the reference set or version be captured? .14
7.1.7 Can external knowledge be applied to the selection of codes? .14
7.2 Data exchange .15
7.2.1 Identification of code systems .15
7.2.2 Identification of code system versions .15
7.2.3 Syntax for post-coordination.15
7.2.4 Presence and representation of translations between code systems .16
7.2.5 Presence of original text .16
7.2.6 Terminology specifications .16
7.3 Data analysis and searching .17
7.3.1 Are subsumed codes included? .17
7.3.2 What mathematical support is expected? .17
7.3.3 How are post-coordinated results handled? .17
7.3.4 How is cross code-system analysis managed? .17
7.3.5 Unknown data .18
8 Sharing and persisting conformance expectations .19
9 Asserting conformance .19
9.1 Conformance and non-conformance .19
9.2 Why conformance statements?.20
9.3 What is conforming? .20
9.4 What is being conformed to? .20
9.5 Assumptions .21
9.6 Support for optional elements.21
9.7 Variations .22
9.8 Completeness .22
10 Evaluating conformance statements .23
11 Verifying conformance .23
11.1 What is verified? .23
11.2 Who verifies? .24
11.3 Can verification be automated? .24
12 Other considerations .24
12.1 Comparing conformance statements .24
12.2 Conformance with conflicting terminology specifications .25
12.3 What IP considerations are associated with the code system? .25
12.4 Terminology services .25
Bibliography .27
iv © ISO 2015 – All rights reserved

Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical Barriers
to Trade (TBT) see the following URL: Foreword - Supplementary information.
The committee responsible for this document is ISO/TC 215, Health informatics.
Introduction
This work item is a Technical Report that will identify and discuss principles and guidelines for the
measurement of conformance in the implementation of terminological systems, in particular, as applied
to Electronic Health Record (EHR) systems.
This item will leverage the current work under way in Canada and will be developed in liaison
with International Health Terminology Standards Development Organization (IHTSDO) and the
Vocabulary Committee of HL7 in the spirit of harmonization across organizations with similar
interests. Additional terminology organizations, active projects and existing expertise will be sought
out for input into this work item.
Conformance is a key step in helping stakeholders determine if implementations of terminology
systems have been done in a correct and consistent manner, particularly as implemented in EHRs.
Loose declarations regarding terminological systems that cannot be tested with meaningful results
do very little to support the end goal of the interoperable EHR. Therefore, the principles and guidelines
for establishing and measuring conformance will focus on identifying the degrees of conformance of
terminological systems with or without use in messaging standards.
This Technical Report is intended to define what is meant by conformance with respect to terminology
systems, particularly as applied to EHR systems, and it is expected to facilitate the formulation of policies
and governance practices locally or nationally. This Technical Report is timely as the emerging IHTSDO
and progressive implementation of the EHR will lead to the increasing awareness of conformance with
respect to terminologies and consistent implementations that allow interoperability by all end-users.
The focus of this Technical Report is to define best practices and a framework for establishing and
measuring conformance. The scope of this Technical Report will include the identification of definitions
and best practice considerations for what constitutes conformance to terminology systems and the
principles for which conformance can be demonstrated.
vi © ISO 2015 – All rights reserved

TECHNICAL REPORT ISO/TR 12310:2015(E)
Health informatics — Principles and guidelines for the
measurement of conformance in the implementation of
terminological systems
1 Scope
The purpose of this Technical Report is to define a framework of good practices for terminology system
maintenance and the principles for which conformance can be demonstrated. The primary focus is the
application of terminology system to Electronic Health Record (EHR) systems, although the principles
and guidelines can be applied broadly in health informatics
The scope of this Technical Report will include, at a minimum, the following considerations for keeping
terminology systems and associated reference material clinically and/or technically relevant and valid:
— governance models and practices;
— high level processes;
— requirements for managing the change.
The scope of this Technical Report will not include a definition of the detailed processes for performing
terminology maintenance.
This Technical Report aims to define the framework of good practices for EHRs and systems regarding
terminology maintenance within these systems. This Technical Report relates directly to the ability
of these records to be safe and legally accurate records of healthcare in the environment of changing
technologies related to the use of clinical terminologies to represent meaning within these systems.
2 Objective
This Technical Report identifies considerations for the expression and evaluation of conformance for
solutions that make use of terminology. The specific focus of this Technical Report is terminology used
in healthcare solutions. However, the principles should apply to solutions implementing terminology
across the health industry. “Solutions” is interpreted broadly and includes both software and hardware
technical implementations, as well as other specifications that are based on or claim to adhere to all or
part of the specification against which conformance is being assessed. Implementation in this Technical
Report does not consider procedural or governance requirements.
By using the definitions and recommendations found here-in, standards bodies, implementers, and
other parties can better achieve their objectives in the development and use of specifications that make
use of terminologies and can better express their terminology capabilities.
This Technical Report is intended to be independent of any particular terminology or terminological
approach, though some portions of the guidance provided will only apply to certain types of terminologies.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
NOTE Because “terminology” is such a broad term, conformance actually needs to be stated in terms of the
various terminology components that are referenced in a specification. These components will also be defined.
3.1
conformance
adherence of a system or specification to the expectations set by another specification
Note 1 to entry: The general definition for conformance has changed over time and been refined for specific
standards. In 1991, ISO/IEC 10641 defined conformance testing as “test to evaluate the adherence or non-
adherence of a candidate implementation to a standard.” ISO/IEC/TR 13233 defined conformance and conformity
as “fulfillment by a product, process or service of all relevant specified conformance requirements.” In recent
years, the term conformity has gained international use and has generally replaced the term conformance in
International Standards.
Note 2 to entry: In 1996 ISO/IEC Guide 2 defined the following three major terms used in this field:
— conformity - fulfillment of a product, process, or service of specified requirements;
— conformity assessment - any activity concerned with determining directly or indirectly that relevant
requirements are fulfilled;
— conformity testing - conformity evaluation by means of testing.
3.2
code system
managed collection of concept representations (3.4) intended for use in persisting or sharing of information
3.3
concept
single mental representation of some real or abstract thing
Note 1 to entry: Concepts should be unique within a code system (3.2).
3.4
concept representation
mechanism by which the system can express a concept (3.3)
Note 1 to entry: Different representations can serve different purposes. Most code systems (3.2) support multiple
representations for each concept (3.3), sometimes even multiple representations of a given type. In some cases,
distinct representations of a concept (3.3) may have their own identifier assigned within the code system (3.2) for
maintenance and internal reference purposes. The types of representations are code (3.4.1), concept id (3.4.2), and
concept designation (3.4.3).
3.4.1
code
concept representation (3.4) intended for use when representing a concept (3.3) in a computable manner
EXAMPLE Passing into a decision support tool or for use in data exchange.
3.4.2
concept id
concept representation (3.4) that is unique within the code system (3.2) and that is used internally by the
code system (3.2) when referencing concepts (3.3)
3.4.3
concept designation
human consumable representation of the concept (3.3)
Note 1 to entry: A concept designation may or may not be a string of characters (could be multimedia); generally
subject to language variants.
2 © ISO 2015 – All rights reserved

3.4.3.1
concept name
concept designation (3.4.3) that is the unique designation of the concept (3.3) in the code system (3.2) and
intended for human understanding
Note 1 to entry: This is usually text but might also be graphical for some code systems (3.2). For example, images
of different facial expressions for a code systems (3.2) representing pain scales.
3.5
code system partition
result of dividing a code identifier namespace into constituent components in order to delegate
responsibility among organizations
Note 1 to entry: Some code systems (3.2) divide their “code” identifier namespace and delegate responsibility for
different sets of codes (3.4.1) to different organizations.
Note 2 to entry: Examples include SNOMED-CT and LOINC. Each delegated organization is then responsible for
the development, maintenance, and publication of the content for its delegated namespace of codes (3.4.1). This
delegation allows organizations to introduce needed codes (3.4.1) more quickly than would be possible with
a centralized approval mechanism. A code system partition is the set of codes (3.4.1) maintained by a single
organization in such a delegated scheme.
3.6
code system supplement
informative extension to a code system (3.2) involving non definitional information to support
implementation
Note 1 to entry: When introducing the use of a code system (3.2), the representations, properties, and relationships
of the concepts (3.3) within that code system (3.2) do not always meet the needs of the potential users of that code
system (3.2). For example, they may require translations of display names or definitions to other languages or using
terminology more familiar to their community. They may need additional properties indicating allowed use of the
concepts (3.3), for example, “Which lab tests are orderable?” These users may choose to “supplement” the code system
(3.2) with additional information so that it meets their requirements. Because no codes (3.4.1) or concepts (3.3) are
added, interoperability based on the underlying code system (3.2) is still maintained. This set of independently
published supplemental information for an existing code system (3.2) is known as a code system supplement.
3.7
local code system
code system (3.2) used only within the organization that maintains the code system (3.2) or in direct
communication with that code system (3.2)
Note 1 to entry: These code systems (3.2) are useful for achieving consistency within an organization but do not
achieve interoperability across organizations. Because they are maintained directly by the organization using
the codes (3.4.1), their maintenance processes are normally very responsive in the addition of new codes (3.4.1);
however, they are frequently not as robust in the following of good vocabulary processes such as avoiding code
(3.4.1) re-use, avoiding overlap, etc.
3.8
terminology binding
assertion of what codes (3.4.1) are to be used at a particular place within a specification, including an
indication of conformance (3.1) expectations
4 Purposes for conformance
For the evaluation of conformance to serve a useful purpose, there has to be some sort of benefit.
There are a variety of benefits to seeking conformance with a specification that includes a terminology
component. This Clause summarizes some of the most frequent objectives.
4.1 Interoperability
One of the most frequent objectives for the enforcement of terminology specifications is to aid
interoperability between systems that use those specifications. Two systems that use different codes
for the same concept or that do not understand the same set of codes are unlikely to interoperate safely.
4.2 Data analysis
Knowledge bases, decision support engines, clinical studies, and other forms of analysis usually
require coded information to be captured in a consistent way. Verifying the conformance of a system or
specification can help to confirm that the data collected using that system will be able to be analysed.
4.3 Consistency of user experience
When users in a particular community will be making use of multiple systems, there are significant
benefits to learning curve and user acceptance if those systems capture data in a consistent way. This
includes the terminologies used to capture data.
4.4 Application functionality
The terminology capabilities of a system or specification are one measure of the sophistication of
that system or specification. A system that is capable of capturing detailed post-coordinated coded
information is significantly more sophisticated than a system that is only capable of capturing free-text
information. As a result, conformance with a terminology specification can be used as a quality measure
independent of any particular expectation of use of the data.
4.5 Acceptance filter
Sometimes, an expectation of conformance may be used as a bar for acceptance in some sort of program.
This might be for a regulatory purpose, as part of a procurement process, or for other reasons. The
expectations might be driven by one or more of the preceding rationales, but it can also just act as a
“filter” to reduce the number of qualifying systems.
5 Conformance process
Once a determination has been made that conformance to a terminology specification will be useful, the
following are four steps to the conformance process.
a) Ensure the specification that conformance is being evaluated against (base specification) clearly
documents terminology expectations.
b) Document the capabilities of the system or specification being evaluated for conformance in terms
of the base specification.
c) Evaluate the documented capabilities against the expectations set by the base specification.
d) If dealing with a system, test the system to verify that the documented capabilities reflect the actual
capabilities.
These steps are discussed in detail in the following subclauses.
5.1 Documenting expectations
Documenting expectations is the most critical step in any terminology conformance exercise. If the base
specification is unclear about requirements, then any evaluation of conformance will be uncertain and
the desired objectives of achieving conformance will not be met. This subclause will discuss areas a
terminology specification should cover in order to be clear and complete.
4 © ISO 2015 – All rights reserved

5.1.1 Optionality
Many specifications allow a degree of flexibility in exactly how terminology (and other aspects of the
specification) needs to be implemented in order to be conformant. This may be due to variations in the
environments in which the specification is being implemented, recognition of reasonable diversity of
implementer capability requirements or needs, or other reasons. It is therefore essential that terminology
specifications clearly differentiate which options are required and which are not. Recommended terms
and definitions are listed below.
5.1.1.1 Optional
Optional requirements are those that may be supported but need not be supported by conformant
systems. The expectation is that any conformant system that does support the requirement will support
the requirement in the manner specified, i.e. the mechanism of implementation is not optional, but the
choice of whether to implement or not is optional. Terminology specifications identifying a requirement
as optional will often use the term “MAY”.
5.1.1.2 Recommended
A “recommended” requirement is an optional requirement where the author of the terminology
specification wishes to assert a best practice. While systems that do not implement a recommended
requirement would technically be conformant, they might still be considered somewhat deficient or
“non-optimal”. Terminology specifications identifying a requirement as recommended will often use the
term “SHOULD”.
5.1.1.3 Required
A “required” portion of a terminology specification shall be implemented by all implementations
that wish to be conformant with the base terminology specification. Failure to implement a required
component automatically renders the solution non-conformant. Terminology specifications identifying
a component as required may use the term “MUST” or “SHALL”.
5.1.1.4 Not recommended
In some cases, a terminology specification will identify what sort of content is not allowed rather than
what is allowed. This can be stated with varying strengths. Non-recommended content is similar to
optional content. The inclusion of the non-recommended content is not non-conformant, but may indicate
that the implementation is “non-optimal”. Terminology specifications identifying a requirement as not
recommended will often use the term “SHOULD NOT”.
5.1.1.5 Not permitted
This is the opposite of “required”. Rather than being non-conformant if the content is not supported,
a system would be non-conformant if it does support non-permitted requirement. Terminology
specifications identifying content as not permitted will often use the term “SHALL NOT”.
5.1.1.6 Conditional
Any of the above degrees of optionality may be conditional. This means that a given part of a terminology
specification might be required, recommended, optional, not recommended, or not permitted based on
some condition. The condition might be based on what other aspects of the terminology specification
are supported, the environment in which the terminology specification are being implemented, or other
factors. Care should be taken when marking components of a terminology specification as “conditional”
that the condition is clearly stated and can be consistently evaluated to either true or false for a given
implementation. As well, the terminology specification shall be clear as to the type of optionality
that applies if the condition is true. For example, is the requirement “conditionally recommended” or
“conditionally required”?
6 Terminology artefact conformance considerations
The following subclauses describe the various questions that should be answered when creating
terminology specifications that reference different types of terminology artefacts. By ensuring the
terminology specification provides clear answers to each of these questions, ambiguity is reduced
enabling conformance to be more accurately assessed.
6.1 Code system considerations
This subclause covers the questions that should be addressed when implementing each code system
referenced by a terminology specification, as well as by terminology specifications that reference
code systems.
6.1.1 What is the code system being referenced?
Ensure that the code system being referred to is clearly identified. For example, “ICD” would not be a
useful reference as there is multiple code systems with that label that have been published over time,
does the author mean ICD9? ICD10? Even specifying ICD10 might not be sufficient as many jurisdictions
have their own versions of the ICD code system that have been supplemented and organized differently.
If the code system is a local code system, it is important that the responsible organization be clearly
distinguished. The set of “lab order types” for one hospital (or even one department) might not be the
same as they are for another, even if they are nearby or even in the same building.
In the event that a terminology specification supplements codes from some non-local code system with
additional local codes, a clear distinction shall be made between the third party codes (which come from
one code system) and the local codes (which come from a distinct code system). This is because updates
and validations may be processed differently against the third party code system than they are against
the local codes.
Reference to a unique identifier assigned by a registry of code systems can help avoid ambiguity when
referring to code systems.
6.1.2 What version(s) of the code system are supported?
Code systems frequently change over time. New codes may be introduced. Older codes may be deprecated.
In some code systems, codes may even be assigned to different code systems over time. Code systems
that follow this practice are extremely risky to use and should be avoided, if possible. Other information
within the code system may also change, such as concept designations, definitions, relationships, etc.
Because of this variability in the content of code systems over time, knowing what version (or versions) of
the code system are expected to be used makes a difference in the expected behaviour of a system. Clear
descriptions of expected behaviour are essential to conformance, therefore identification of the version
or versions of a code system that are supported is essential when defining terminology conformance.
Code system authors may define different mechanisms for identifying versions of the code system. Some
code system authors may not recognize the concept of version at all. They simply change the contents of the
code system as and when they need to or wish to. Even among those code systems that do assign version
identifiers, the consistency with which versioning occurs may vary. For example, some code system
authors may still apply changes within a version, making the maintainer-assigned version id insufficient,
or at least sub-optimal for identifying exactly what set of code system content is being referred to.
Because of the non-reliability or even non-existence of code system author assigned version identifiers,
a date stamp or even a timestamp may be the most effective mechanism for explicitly referring to a
particular version of many code systems. If taking this approach, the terminology specification should
be clear on exactly what the date means, is it the date the content is changed, the date the content is
published, or the date the content is “intended for use”?
Where using a terminology maintenance organization-assigned version identifier, care should be
taken to ensure consistency in the representation of the identifier with respect to the capitalization,
6 © ISO 2015 – All rights reserved

whitespace, punctuation, and other features that can occasionally interfere with matching of string-
based identifiers.
6.1.3 How are codes represented?
Codes are only useful if they are captured consistently and can be recognized by systems searching and
manipulating them. In some cases, the display and persistence of coded values can vary. Terminology
specifications shall identify rules around how codes must be expressed for their various uses. Possible
issues that can interfere with consistent representation include the following.
— Case sensitivity: Are code representations case-sensitive or case-insensitive? If case-insensitive, is
there a convention of always using upper case or lower-case?
— Character set: Does the code system make use of special characters that are only available in limited
or specialized character sets?
— Whitespace: Is whitespace significant in determining matches for codes? This can impact how
codes are stored.
— Punctuation and special characters: Are the codes typically displayed with separator punctuation
and, if so, is this punctuation significant in distinguishing between codes? Is there a convention that
the punctuation must be present or absent when persisting or displaying to users?
— Leading zeroes: For “numeric” codes, are leading zeroes present on some coded values, and if so,
are they significant? E.g. is “001” equivalent to “1”? Is there a convention for whether leading zeroes
should be present or absent, and if present, how many digits should be present?
6.1.4 What are the version migration expectations, if any?
In the event that the terminology specification does not limit itself to the support of a single version or
explicit set of versions, expectations need to be established for the timeliness of support for new versions
once they are made available. For example, if the code system authority publishes a new version of a code
system on a given day, is an implementation non-conformant if it fails to process codes appropriately
based on that new version an hour later? What about three years later?
Failure to set expectations on the timeliness in which implementations of a terminology specification
support new versions of the code systems referenced by the terminology specification will obviously
affect consistency of implementation and make conformance evaluation challenging. This issue can
be reduced by code systems that publish changes with “future-dated” effective dates. This allows the
changes to be distributed, implemented, and tested prior to the change actually being enabled.
Even when expectations are clear, in the absence of a “future-dating” code system, implementation
realities will often make a synchronized “instantaneous” transition of all implementations of a
terminology specification unfeasible. Terminology specifications that deal with interoperability should
therefore ensure that policies support interoperability even where the communicating systems do not
receive the new version of a code system at the same time. For example,
— supporting the receipt of “deprecated” codes throughout the allowed migration window, and
— either tolerating the receipt of unrecognized codes or (even better) automatically checking for an
update to the code system upon receipt of an unrecognized code.
6.1.5 What is the authoritative source that will be used for processing updates to a given
terminology?
Some code systems have a single authoritative distribution system, be it a terminology service, website
or other publication mechanism. However, many other code systems and updates to those code systems
may be distributed from a variety of sources. For example, ISO code systems may be distributed by
authorized national standards bodies. SNOMED CT updates may be distributed through their respective
country affiliates, etc.
Not all of these various distribution mechanisms will necessarily be synchronous. That means that two
different systems each looking at the most “current” version of a code system from their respective
distribution source may be looking at different versions of a code system. In some cases, the distribution
of code systems may even accidentally introduce discrepancies, meaning there could be unintentional
variations between the same versions of a code system from different distribution sources.
To ensure maximum consistency between implementations, it is therefore recommended that
terminology specifications referencing a code system also indicate the authoritative source for that
code system from the perspective of conformance with that sys
...