ISO/TS 17117:2002

TECHNICAL ISO/TS
SPECIFICATION 17117
First edition
2002-02-15
Health informatics — Controlled health
terminology — Structure and high-level
indicators
Informatique de santé — Terminologie contrôlée relative à la santé —
Structure et indicateurs de haut niveau

Reference number
©
ISO 2002
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not
be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this
file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this
area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters
were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event
that a problem relating to it is found, please inform the Central Secretariat at the address given below.

©  ISO 2002
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic
or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body
in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.ch
Web www.iso.ch
Printed in Switzerland
ii © ISO 2002 – All rights reserved

Contents Page
Foreword.iv
Introduction.v
1 Scope .1
2 Normative references.1
3 Terms and definitions .2
4 General.3
4.1 Basics .3
4.2 Concept orientation.3
4.3 Purpose and scope.4
4.4 Mapping .4
4.5 Systematic definitions.4
4.6 Formal definitions.4
4.7 Explicitness of relations .5
4.8 Reference terminologies.5
4.9 Atomic reference terminologies.5
4.10 Colloquial terminologies.5
5 Structure of the terminology model.5
5.1 Terminology structures.5
5.2 Compositional terminologies .5
6 Maintenance .8
6.1 Basics .8
7 Evaluation.9
7.1 Basics .9
Annex A (informative) History of classification .14
Annex B (normative) Principles for implementation .15
Bibliography.23

Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies (ISO
member bodies). The work of preparing International Standards is normally carried out through ISO technical
committees. Each member body interested in a subject for which a technical committee has been established has
the right to be represented on that committee. International organizations, governmental and non-governmental, in
liaison with ISO, also take part in the work. ISO collaborates closely with the International Electrotechnical
Commission (IEC) on all matters of electrotechnical standardization.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 3.
The main task of technical committees is to prepare International Standards. Draft International Standards adopted
by the technical committees are circulated to the member bodies for voting. Publication as an International
Standard requires approval by at least 75 % of the member bodies casting a vote.
In other circumstances, particularly when there is an urgent market requirement for such documents, a technical
committee may decide to publish other types of normative document:
— an ISO Publicly Available Specification (ISO/PAS) represents an agreement between technical experts in an
ISO working group and is accepted for publication if it is approved by more than 50 % of the members of the
parent committee casting a vote;
— an ISO Technical Specification (ISO/TS) represents an agreement between the members of a technical
committee and is accepted for publication if it is approved by 2/3 of the members of the committee casting a
vote.
An ISO/PAS or ISO/TS is reviewed after three years with a view to deciding whether it should be confirmed for a
further three years, revised to become an International Standard, or withdrawn. In the case of a confirmed ISO/PAS
or ISO/TS, it is reviewed again after six years at which time it has to be either transposed into an International
Standard or withdrawn.
Attention is drawn to the possibility that some of the elements of this Technical Specification may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights.
ISO/TS 17117 was prepared by Technical Committee ISO/TC 215, Health informatics.
Annex B forms a normative part of this Technical Specification. Annex A is for information only.

iv © ISO 2002 – All rights reserved

Introduction
In 1839, William Farr stated in his First Annual Report of the Registrar-General of Births, Deaths and Marriages in
England, “The nomenclature is of as much importance in this department of inquiry, as weights and measures in
the physical sciences, and should be settled without delay.” Since that time, this theme has been heard resounding
from an increasingly large group of scientists (see Annex A). Today, the need for controlled terminologies to
support health record systems has been widely recognized (E-1238, E-1239, E-1384, E-1633, ENV 12017).
Controlled terminologies provide systems with the means to aggregate data. This aggregation of data can be
carried out at multiple levels of granularity and therefore can enhance the clinical retrieval of a problem-oriented
record, data pertaining to a classification for billing purposes, or outcomes data for a given population. Maintenance
of large-scale terminologies has become a burdensome problem, as the size of term sets has escalated. Without a
well-structured backbone, large-scale terminologies cannot scale to provide the level of interoperability required by
today’s complex electronic health record applications.
[7]
The solution rests with standards . Over the past ten or more years, medical informatics researchers have been
studying controlled terminology issues directly. They have examined the structure and content of existing
terminologies to determine why they seem unsuitable for particular needs, and they have proposed solutions. In
[8]
some cases, proposed solutions have been carried forward into practice and new experience has been gained.
As we have entered the twenty-first century, it seems appropriate to pause to reflect on this experience, and to
publish a standard set of goals for the development of comparable, reusable, multipurpose and maintainable
controlled health terminologies (ISO 12200, ISO 12620).
This Technical Specification is the first deliverable for the ISO/TC 215 Health informatics, Working group 3, Health
concept representation, that is also working on an International Standard to be the basis for future standards in this
area. It will serve as a guide for governments, funding agencies, terminology developers, terminology integration
organizations and the purchasers and users of controlled health terminology systems toward improved
terminological development and recognition of value in a controlled health terminology. This ISO/TS 17117 on
quality indicators of controlled health terminologies is based on previous work in ASTM that naturally could not be
harmonized with ISO work already in progress. The present work is therefore published as a Technical
Specification at this time with the intent to revise it to be compatible with the planned basic terminology standard
and converted to a full International Standard after a maximum of three years.

TECHNICAL SPECIFICATION ISO/TS 17117:2002(E)

Health informatics — Controlled health terminology — Structure
and high-level indicators
1 Scope
This Technical Specification specifies the principal ideas which are necessary and sufficient to assign value to a
controlled health terminology. It is applicable to all areas of healthcare about which information is kept or utilized.
2 Normative references
The following normative documents contain provisions which, through reference in this text, constitute provisions of
this Technical Specification. For dated references, subsequent amendments to, or revisions of, any of these
publications do not apply. However, parties to agreements based on this Technical Specification are encouraged to
investigate the possibility of applying the most recent editions of the normative documents indicated below. For
undated references, the latest edition of the normative document referred to applies. Members of ISO and IEC
maintain registers of currently valid International Standards.
ISO 704, Terminology work — Principles and methods
ISO 860, Terminology work — Harmonization of concepts and terms
ISO 1087-1, Terminology work — Vocabulary — Part 1: Theory and application
ISO 1087-2, Terminology work — Vocabulary — Part 2: Computer applications
ISO/IEC 11179-3, Information technology — Specification and standardization of data elements — Part 3: Basic
attributes of data elements
ISO 12620, Computer applications in terminology — Data categories
ISO/IEC 2382-4, Information technology — Vocabulary — Part 4: Organization of data
ISO/IEC/TR 9789, Information technology — Guidelines for the organization and representation of data elements
for data interchange — Coding methods and principles
E-1284, Standard Guide for Construction of a Clinical Nomenclature for Support of Electronic Health Records
E-1712, Standard Specification for Representing Clinical Laboratory Test and Analyte Names
ENV 12264, Health Informatics — Categorical Structures of Systems of Concepts — Model for Representation of
Semantics
3 Terms and definitions
For the purposes of this Technical Specification, the following terms and definitions apply.
3.1
terminology
set of terms representing a system of concepts within a specified domain
NOTE This implies a published purpose and scope from which one can determine the degree to which this representation
adequately covers the domain specified.
3.2
controlled health terminology
set of terms intended for clinical use
NOTE This implies enough content and structure to provide a representation capable of encoding comparable data, at a
granularity consistent with that generated by the practice within the domain being represented, within the purpose and scope of
the terminology.
3.3
classification
terminology which aggregates data at a prescribed level of abstraction for a particular domain
NOTE 1 This fixing of the level of abstraction that can be expressed using the classification system is often done to enhance
consistency when the classification is to be applied across a diverse user group, such as is the case with some of the current
billing classification schemes.
NOTE 2 See Annex A for the history of classification.
3.4
ontology
organization of concepts for which a rational argument can be made
EXAMPLE A hierarchy of qualifiers would be a qualifier ontology.
NOTE Colloquially, this term is used to describe a hierarchy constructed for a specific purpose.
3.5
qualifier
string which, when added to a term, changes the meaning of the term in a temporal or administrative sense
EXAMPLES “History of ” or “recurrent”.
3.6
modifier
string which, when added to a term, changes the meaning of the term in the clinical sense
EXAMPLES “Clinical stage” or “severity of illness”.
3.7
canonical term
preferred atomic or pre-coordinated term for a particular medical concept
3.8
term
word or words corresponding to one or more concepts
2 © ISO 2002 – All rights reserved

4 General
4.1 Basics
The basic characteristics of a terminology influence its utility and appropriateness in clinical applications.
Terminologies should be evaluated within the context of their stated scope and purpose and are intended to
complement and utilize those notions already identified by other national and international standards bodies.
This Technical Specification explicitly refers only to terminologies that are primarily designed to be used for clinical
concept representation or to the aspect of a terminology designed to be used for clinical concept representation.
This Technical Specification will also provide terminology developers and authors with the quality guidelines
needed to construct useful and maintainable controlled health terminologies. These tenets do not attempt to specify
all the richness which can be incorporated into a health terminology. However, this Technical Specification does
specify the minimal requirements, which, if not adhered to, will assure that the terminology will have only limited
generalizability and will be very difficult, if not impossible, to maintain. Terminologies which do not currently meet
these criteria, can be in compliance with this Technical Specification by putting in place mechanisms to move
toward these goals. Principles for implementation are specified in Annex B.
This Technical Specification will provide terminology developers with a sturdy starting point for the development of
controlled health terminologies. This foundation serves as the basis from which terminology developers will build
robust, large-scale, reliable and maintainable terminologies.
4.2 Concept orientation
The basic unit of a terminology shall be a concept, which is the embodiment of some specific meaning and not a
code or character string. Identifiers of a concept shall correspond to one and only one meaning and, in a well-
ordered terminology, only one concept may have that same meaning, as specified in ISO 860. However, multiple
terms (linguistic representations) may have the same meaning if they are explicit representations of the same
concept. This implies non-redundancy, non-ambiguity, non-vagueness and internal consistency.
4.2.1 Non-redundancy
Terminologies shall be internally normalized. There shall not be more than one concept identifier in the terminology
with the same meaning, as specified in ISO 704 and E-1284. This does not exclude synonymy, rather it requires
that this be explicitly represented.
4.2.2 Non-ambiguity
No concept identifier should have more than one meaning. However, an entry term can point to more than one
concept.
EXAMPLE MI as myocardial infarction and mitral insufficiency.
NOTE Some authors have referred to entry terms as an interface terminology.
4.2.3 Non-vagueness
Concept names shall be context free.
EXAMPLE “Diabetes mellitus” should not have the child concept “well controlled”, instead the child concept's name should
be “diabetes mellitus, well controlled”.
NOTE Some authors have referred to context free as context laden.
4.2.4 Internal consistency
Relationships between concepts should be uniform across parallel domains within the terminology.
EXAMPLE If heart valve structures are specified anatomically, the diagnosis related to each structure is also specified using
the same relationships.
4.3 Purpose and scope
Any controlled terminology shall have its purpose and scope clearly stated in operational terms so that its fitness
for particular purposes can be assessed and evaluated. Where appropriate, it may be useful to illustrate the scope
by examples or “use cases” as in database models and other specification tools. Criteria, such as coverage and
comprehensiveness, can only be judged relative to the intended use and scope.
EXAMPLE A terminology might be comprehensive and detailed enough for general practice with respect to cardiovascular
signs, symptoms and disorders, but inadequate to a specialist cardiology or cardiothoracic surgery unit. Conversely, a
terminology sufficiently detailed to cope with cardiology and cardiothoracic surgery might be totally impractical in general
practice.
4.3.1 Coverage
Each segment of the healthcare process shall have explicit in-depth coverage, and not rely on broad leaf node
categories that place specific clinical concepts together. The extent to which the depth of coverage is incomplete
[9]
shall be explicitly specified for each domain (scope) and purpose as indicated in 4.3.
EXAMPLE It is often important to distinguish specific diagnosis from categories presently labelled “not elsewhere classified”
(NEC), or to differentiate disease severity such as indolent prostate cancer from widely metastatic disease.
4.3.2 Comprehensiveness
The extent to which the degree of comprehensiveness is incomplete shall be explicitly specified for each domain
(scope), and purpose as indicated in 4.3. Within the scope and purpose, all aspects of the healthcare process shall
be addressed for all related disciplines, such as physical findings, risk factors, or functional status — across the
breadth of medicine, surgery, nursing and dentistry. This criterion applies because decision support, risk
adjustment, outcomes research and useful guidelines require more than diagnoses and procedures.
EXAMPLES The existing Agency for Healthcare Research and Quality Guidelines, and the Healthcare Finance Administration
[10]
(HCFA) mortality model.
4.4 Mapping
Government and payers mandate the form and classification schema for much clinical data exchange. Thus,
comprehensive and detailed representations of patient data within computer-based patient records should have the
ability to be mapped to those classifications, such as ICD-9. This need for multiple granularities is required for
clinical healthcare, as well as is specified in ISO/IEC/TR 9789. The degree to which the terminology is mappable to
[11]
other classifications shall be explicitly stated.
EXAMPLE An endocrinologist may specify more detail about a patient’s diabetes mellitus than a generalist working in a
primary care setting, even though both specialities may be caring for the same patient.
4.5 Systematic definitions
In order for users of the terminology to be certain that the meaning that they assign to concepts is identical to the
meaning which the authors of the terminology have assigned to these definitions will need to be explicit and
available to the users. Further, as relationships are built into terminologies, multiple authors will need these
definitions to ensure consistency in authorship.
EXAMPLE The clinical concept “hypertension” might be defined as a consistently elevated blood pressure and needs to be
distinguished from a single “BP > 140/85”.
4.6 Formal definitions
A compositional system should contain formal definitions for non-atomic concepts and formal rules for inferring
subsumption from the definitions, as specified in E-1712.
4 © ISO 2002 – All rights reserved

4.7 Explicitness of relations
The logical definition of subsumption should be defined. The formal behaviour of all links/relations/attributes should
be explicitly defined. If a looser meaning such as “broader than/narrower than” is used, it should be explicitly
stated.
EXAMPLE The primary hierarchical relation should be subsumption as exemplified by logical implication: B is a kind of A
means all Bs are As.
4.8 Reference terminologies
The set of canonical concepts, their structure, relationships and, if present, their systematic and formal definitions
define the core of the controlled health terminology.
4.9 Atomic reference terminologies
In a reference terminology consisting of only atomic concepts and their systematic definitions, no two or more
concepts can be combined to create a composite expression which has the same meaning as any other single
concept contained in the atomic reference terminology.
4.10 Colloquial terminologies
The set of terms, which consists of commonly used entry points, maps to one or more canonical terms within the
terminology.
NOTE These have been called “entry terms” or “interface terminologies” by different authors.
5 Structure of the terminology model
5.1 Terminology structures
Terminology structures determine the ease with which practical and useful interfaces, for term navigation, entry, or
retrieval can be supported, as specified in ISO 704, ISO 1087-1 and ENV 12264.
5.2 Compositional terminologies
5.2.1 Compositionality
Composite concepts are created from atomic and pre-coordinated concepts and shall have the ability to be
[12]
combined to create compositional expressions .
EXAMPLE “Colon cancer” comprises “malignant neoplasm” and “large bowel” as atomic components. In a compositional
system, concept representations can be divided into atomic and composite concept representations.
Composite concept representations can be further divided into named “pre-coordinated concept representations”
and “post-coordinated representation” expressions. Within a composite concept, it may be possible to separate the
constituents into three categories: “kernel concept”, “qualifier (also called “status”) concept”, and “modifier concept”.
NOTE A concept is a notion represented by language, which identifies one idea. However, the term “concept” in this
Technical Specification is used to refer to the representation of a concept rather than to the thought itself.
5.2.1.1 Atomic concept
An atomic concept is a representation of a concept that is not composed of other simpler concept representations
within a particular terminology. In many cases, atomic concepts will correspond to what philosophers call “natural
kinds”. Such an entity cannot be meaningfully decomposed. Concepts should be separable into their constituent
components, to the extent practical. These should form the root basis of all concepts.
EXAMPLE In SNOMED-RT, “colon” is a synonym for “large bowel” and “cancer” is a synonym for “neoplasm, malignant”.
Therefore, the term “colon cancer” is non-atomic as it can be broken down into “large bowel” and “neoplasm, malignant”. Each
of these two atomic terms has a separate and unique concept identifier, as does the pre-coordinated term “colon cancer”.
5.2.1.2 Composite concept
A composite concept is composed as an expression made up of atomic concepts linked by semantic relations
(such as roles, attributes or links).
5.2.1.2.1 Pre-coordinated concept
Such an entity can be broken into parts without loss of meaning (can be meaningfully decomposed), when the
atomic concepts are examined in aggregate. These are representations, which are considered single concepts
within the host vocabulary. Ideally, these concepts should have their equivalent composite concepts explicitly
defined within the terminology (that is the terminology should be normalized for content, as indicated in 5.2.2).
EXAMPLE The term “colon cancer” is non-atomic, however it has a single unique identifier, which means to the
SNOMED-RT that it represents a “single” concept. It has the same status in the terminology as the site “large bowel” and the
diagnosis “neoplasm, malignant”.
5.2.1.2.2 Post-coordinated concept
A post-coordinated concept is a composite concept, which is not pre-coordinated and therefore shall be
represented as an expression of multiple concepts using the representation language. This is the attempt of a
system to construct a set of concepts from within a controlled terminology to more completely represent a user’s
query.
EXAMPLE The concept “bacterial effusion, left knee” is not a unique term within the SNOMED-RT terminology. It represents
a clinical concept that some patient has an infected left knee joint. As it cannot be represented by a single concept identifier, to
fully capture the intended meaning a system would need to build a representation from multiple concept identifiers or lose
information to free text.
5.2.1.3 Types of atomic and pre-coordinated concepts
Unique concept representations can be classified within a terminology into at least three distinct types: kernel
concepts, modifiers and qualifiers (which contain status concepts). This separation allows user interfaces to provide
more readable and therefore more useful presentations of composite concepts.
5.2.1.3.1 Kernel concept
This is an atomic or pre-coordinated concept, which represents one of the one or more main concepts within a pre-
coordinated or post-coordinated composition.
5.2.1.3.2 Modifiers and qualifiers
Constituents of a composite concept that refine the meaning of a kernel concept are known as modifiers or
qualifiers.
EXAMPLE 1 “Stage 1a” in the expression “having colon cancer stage 1a” or “brittle, poorly controlled” in the expression
“brittle, poorly controlled diabetes mellitus” are examples of qualifiers and modifiers.
6 © ISO 2002 – All rights reserved

In general, these concepts are expressed as a link plus a value (“attribute-value-pair”). Terminologies shall support
a logical structure that can support temporal duration and trend. Attributes shall be themselves elements of a
terminology, and fit into a practical model that extends a terminology.
EXAMPLE 2 Cancers may be further defined by their stage and histology, have been symptomatic for a specifiable time, and
may progress over a given interval.
Attributes are required to capture important data features for structured data entry and pertinent to secondary data
uses such as aggregation and retrieval. Kernal concepts can be refined in many ways, including a clinical sense, a
temporal sense and by status terms, such as “recurrent”.
5.2.2 Normalization of content
Normalization is the process of supporting and mapping alternative words and shorthand terms for composite
concepts. All pre-coordinated concepts shall be mapped to or logically recognizable by all possible equivalent post-
coordinated concepts. There should be mechanisms for identifying this synonymy for user created (new) post-
coordinated concepts as well (i.e. when there is no pre-coordinated concept for this notion in the terminology). This
functionality is critical to define explicitly equivalent meaning, and to accommodate personal, regional and discipline
specific preferences. Additionally, the incorporation of terms as synonyms, represented in a language other than
that primarily used in the host terminology, can achieve a simple form of multilingual support.
5.2.3 Normalization of semantics
In compositional systems, there exists the possibility of representing the same concept with multiple potential sets
of atoms, which may be linked by different semantic links. In this case, the terminology needs to be able to
recognize this redundancy/synonymy (depending on your perspective). The extent to which normalization can be
performed formally by the system should be clearly indicated.
EXAMPLE The concept represented by the term “Laparoscopic Cholecystectomy” might be represented in the following two
dissections:
Surgical Procedure: Excision”{Has Site Gallbladder}, {Has Method Endoscopic}
and
Surgical Procedure: Excision”{Has Site Gallbladder}, {Using Device Endoscope}.
5.2.4 Multiple hierarchies
Concepts should be accessible through all reasonable hierarchical paths, i.e they shall allow multiple semantic
parents. A balance between number of parents (as siblings) and number of children in a hierarchy should be
maintained. This feature assumes obvious advantages for natural navigation of terms (for retrieval and analysis),
as a concept of interest can be found by following intuitive paths (i.e. users should not have to guess where a
[13]
particular concept was instantiated).
EXAMPLE One example of multiple semantic parentage is “stomach cancer” which can be viewed as a “neoplasm” or as a
“gastrointestinal disease”.
5.2.5 Consistency of view
A concept in multiple hierarchies shall be the same concept in each case. The previous example of stomach
cancer in 5.2.4 shall not have changes in nuance or structure when arrived at via the cancer hierarchy as opposed
to the gastrointestinal disease hierachy. Inconsistent views could have catastrophic consequences for retrieval and
decision support, by inadvertently introducing variations in meaning which may be unrecognized and therefore be
[14]
misleading to users of the system.
5.2.6 Explicit uncertainty
Notions of “probable”, “suspected”, “history of” or differential possibilities such as a differential diagnosis list, shall
be supported. The impact of “certain” versus “very uncertain” information has obvious impact on decision support
and other secondary data uses. Similarly, in the case of incomplete syndromes, clinicians should be able to record
the partial criteria consistent with the patient’s presentation. These criteria are listed separately, as many current
terminological systems fail to address this area adequately.
5.2.7 Representational form
The representational form of the identifiers within the terminology should be meaningless. Computer coding of
concept identifiers shall not place arbitrary restrictions on the terminology, such as numbers of digits, attributes, or
composite elements. To do so subverts meaning and content of a terminology to the limitations of format, which, in
turn, often results in the assignment of concepts to the wrong location because it might no longer “fit” where it
belongs in a hierarchy. These reorganizations confuse people and machines alike, as intelligent navigation agents
are led astray for arbitrary reasons. The long, sequential, alphanumeric tags used as concept identifiers in the
UMLS project of the National Library of Medicine exemplify well this principle.
6 Maintenance
6.1 Basics
Technical choices can impact the capacity of a terminology to evolve, change and remain usable over time.
6.1.1 Context-free identifiers
Unique codes attached to concepts shall not be tied to hierarchical position or other contexts; their format shall not
carry meaning. Because health knowledge is being constantly updated, the categorization of health concepts is
likely to change. For this reason, the “code” assigned to a concept shall not be inextricably bound to a hierarchy
position in the terminology, so that the code need not change when concepts are hierarchically reorganized.
Changing the “code” may make historical patient data confusing or erroneous.
EXAMPLE “Peptic ulcer disease” is now understood as an infectious disease, but this was not always so.
[15]
NOTE This notion of context-free identifiers is the same as non-semantic identifiers.
6.1.2 Persistence of identifiers
Codes shall not be reused when a concept is obsolete or superseded. Consistency of patient description over time
is not possible when concepts change codes; the problem is worse when codes can change meaning. This practice
not only disrupts historical analyses of aggregate data, but can be dangerous to the management of individual
patients whose data might be subsequently misinterpreted.
NOTE This encompasses the notion of concept permanence.
6.1.3 Version control
Updates and modifications shall be referable to consistent version identifiers. Usage in patient records should carry
this version information. This is true because the interpretation of coded patient data is a function of terminologies
that exist at a point in time.
EXAMPLE AIDS patients were coded inconsistently before the introduction of the term AIDS.
Terminology representations should specify the state of the terminology system at the time a term is used; version
information most easily accomplishes this, and may be hidden from ordinary review, as specified in ISO 12620,
[16], [17]
ISO 1087-2, ISO 11179-3 and ISO/IEC 2382-4.
8 © ISO 2002 – All rights reserved

6.1.3.1 Editorial information
New and revised terms, concepts and synonyms shall have their date of entry or effect in the system, along with
pointers to their source and/or authority. Previous ways of representing a new entry should be recorded for
historical retrieval purposes.
6.1.3.2 Obsolete marking
Superseded entries should be so marked, together with their preferred successor. Because data may still exist in
historical patient records using obsolete terms, their future interpretation and aggregation are dependent upon that
term being carried and cross-referenced to subsequent terms.
EXAMPLE Human T-cell leukemia virus — type III (HTLV III) to human immunodeficiency virus (HIV).
6.1.4 Recognize redundancy
Authors of these large-scale terminologies will need mechanisms to identify redundancy when it occurs. This is
essential for the safe evolution of any such terminology. This implies normalization of concepts and semantics (see
5.2.2 and 5.2.3), but specifically addresses the need for terminology systems to provide the tools and resources
necessary to accomplish this task.
6.1.5 Language independence
It would be desirable for terminologies to support multilingual presentations. As healthcare confronts the global
economy and multiethnic practice environments, routine terminology maintenance shall incorporate multilingual
support. While substantially lacking the power and utility of machine translation linguistics, this simplistic addition
will enhance understanding and use globally. Have there been translations? What is the expected cost of
translation?
6.1.6 Responsiveness
The frequency of updates, or sub-versions, should be sufficiently short to accommodate new codes and repairs
quickly, ideally in the order of weeks.
7 Evaluation
7.1 Basics
As we seek to understand quality in the controlled terminologies that are created or used, a standard criterion for
the evaluation of these systems is needed. All evaluations shall reflect and specifically identify the purpose and
[18]
scope of the terminology being evaluated.
7.1.1 Measures of purpose and scope
Important dimensions along which purpose and scope should be defined to address the following issues.
7.1.1.1 Clinical area
What is the clinical area of use of the terminology, the disease area of patients addressed and/or the expected
profession of users? Within what parts of healthcare is the terminology intended to be used and by whom?
7.1.1.2 Primary use
What is the primary intended usage of the terminology?
EXAMPLE Some areas of usage include: reporting for remuneration, management planning, epidemiological research,
indexing for bibliographic, web-based retrieval, recording of clinical details for direct patient care, use for decision support,
linking of record to decision support, etc.
7.1.1.3 Persistence and extent of use
While some terminologies are intended, at least initially, primarily for a specific study or a specific site, others are
not. If intended to be persistent, what are the means of updating or change management, etc.?
7.1.1.4 Degree of automatic inferencing intended
Developers should define whether or not and to what degree automatic inferencing is intended. Developers should
define whether or not classification is intended to be automatic. Developers should define whether or not it is
intended that validation on input be possible and within what limits. Developers should define whether or not post-
coordinated expressions are to be accepted and if so, what can be inferred about them and what restrictions shall
be placed on them (i.e. is formal sanctioning required?).
7.1.1.5 Transformations (mappings) to other terminologies
What transformations (mappings) are supported and for what intended purpose? What is the sensitivity and
specificity of the transformations?
EXAMPLE Transformation for purposes of bibliographic retrieval may require less precision than transformation for clinical
usage.
7.1.1.6 User/developer extensibility
Is it intended that the terminology be extended by users or application developers? If so, within what limits? If not,
what mechanisms are available for meeting new needs as they arise?
7.1.1.7 Natural language
Is natural language input or output supported (for analysis or input)? To what level of accuracy?
7.1.1.8 Other functions
What other functions are intended?
EXAMPLE Linkage to specific decision support systems, linkage to post-marketing surveillance, etc.
7.1.1.9 Current status
To what extent is the system intended to be “finished” or work in progress? If different components of the
terminology are at different stages of completion, how is this indicated?
7.1.2 Measures of quality, terminological tools
7.1.2.1 Interconnectivity (mapping)
7.1.2.1.1 Terminology and other coding systems
To what extent is the terminology mappable to other coding systems or reference terminologies?
7.1.2.1.2 Terminology and terminological enhancements
To what extent can the terminology accommodate local terminological enhancements?
10 © ISO 2002 – All rights reserved

7.1.2.1.3 Terminology and networking
Can the terminology server respond to queries sent over a network (LAN, WAN)?
7.1.2.2 Precision and recall
7.1.2.2.1 Terminology
What are the terminology's precision and recall for mapping diagnoses, procedures, manifestations, anatomy,
organisms, etc. against an established and nationally recognized standard query test set, using a standard, well-
principled method? This should be evaluated only within the intended scope and purpose of the terminology
system.
7.1.2.2.2 Search engine
Is a standard search engine used in the mapping process?
7.1.2.3 Usability
7.1.2.3.1 Validation of usability
Has the usability of the terminology been verified?
7.1.2.3.2 Interface considerations
How have interface considerations been separated from terminology evaluation?
7.1.2.3.3 Prototypes
Has an effective user interface been built? Has the terminology been shown to have an effective user interface for
its intended use? If not, what are the questions or issues outstanding? Is there evidence for speed of entry,
accuracy, comprehensiveness in practice etc. with different approaches? If not, is there proof of concept?
7.1.2.3.4 Application programmer interfaces
Is there support for computer interfaces and system implementers? Is there demonstrated proof of concept
implementation in software? Can it be shown to be usable for the primary purpose indicated? Have there been
failed implementations?
7.1.2.4 Feasibility
If the terminology is intended for use in an electronic patient record (EPR), what are the options for information
storage? Has feasibility been demonstrated?
7.1.3 Other measures of quality
The generalizability (applicability) of any study design reported (evaluating reported evaluations) should have the
ability to be evaluated.
7.1.3.1 Healthcare/clinical relevance
What is the terminology's healthcare/clinical relevance?
7.1.3.2 Gold standard
What was the gold standard used in the evaluation?
7.1.3.3 Study population
If published population rates are used for comparison, was the study population comparable to the population from
which the rates were derived?
7.1.3.4 Specific aims
Were the specific aims clear?
7.1.3.5 Blinding
Was the study appropriately blinded?
7.1.3.6 Randomization
Was the test set selection randomized or shown in some sense to be a representative sample of the end user
population?
7.1.3.7 Test location
7.1.3.7.1 Independence
Was the test location different from the developer’s location?
7.1.3.7.2 Appropriate for study design
How was the test site suited to the study design (tools, resources, etc.)?
7.1.3.7.3 Principal investigator associations
Was the principal investigator associated with a:
 university;
 academic medical centre;
 corporation or company;
 hospital;
 government agency;
 primary care centre (health maintenance organization);
 private practice;
 academic organization?
7.1.3.7.
...