ISO/TS 18864:2017

TECHNICAL ISO/TS
SPECIFICATION 18864
First edition
2017-11
Health informatics — Quality metrics
for detailed clinical models
Informatique de santé — Indicateurs de qualité pour modèles
cliniques détaillés
Reference number
©
ISO 2017
© ISO 2017, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO 2017 – All rights reserved

Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms . 5
5 Relationship with other activities . 6
5.1 ISO/TS 13972 . 6
5.2 ISO 13606-2 . 6
5.3 EuroRec quality criteria for archetypes . 6
5.4 Delphi study . 6
6 The purpose and importance of DCM quality . 6
7 Quality aspect of DCMs . 7
8 DCM quality framework . 7
8.1 DCM quality framework component . 7
8.2 DCM Quality characteristics . 8
8.3 DCM Quality metrics . 8
8.4 Expected benefits of high quality DCMs . 9
9 Quality metrics for DCMs . 9
9.1 Design and development . 9
9.1.1 General. 9
9.1.2 Stakeholders participated in the development (or design) of DCM .10
9.1.3 Stakeholders participated in the verification/approval of DCM .10
9.1.4 Translations, only if applicable .10
9.1.5 Relationships .11
9.2 Per detailed clinical model .11
9.2.1 Compliance to standard .11
9.3 Metadata .13
9.3.1 DCM Version .13
9.3.2 Purpose of DCM .13
9.3.3 Appropriate description of application target patient or population of DCM .14
9.3.4 Description of multiple uses .14
9.3.5 Appropriate description of discipline of DCM user .15
9.3.6 DCM author(s) .15
9.3.7 Authoring date .15
9.3.8 Modification date .15
9.3.9 Initial review round date .16
9.3.10 Number of review rounds .16
9.3.11 Last review round date . .16
9.3.12 Status of content publication .17
9.3.13 Mention of reference(s) used in DCM development, only if applicable .17
9.4 Per data element .17
9.4.1 Valid value of DCM .17
9.4.2 Terminology binding .18
9.5 Governance .18
9.5.1 Maintenance organization of DCM .18
9.5.2 Existence of user feedback mechanism for DCM .18
9.5.3 Realm-specific specializations and extensions .19
9.5.4 Multiple outputs .19
9.5.5 Clinicians participated in the verification/validation of DCM for
determining customer satisfaction .19
Annex A (informative) Some published quality requirements and criteria for DCMs .21
Annex B (informative) Comparison of Detailed Clinical Modelling approach .24
Bibliography .28
iv © ISO 2017 – All rights reserved

Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following
URL: www.iso.org/iso/foreword.html.
This document was prepared by Technical Committee ISO/TC 215, Health informatics.
Introduction
Normalization and formalization of evidence-based clinical data are fundamental requirements towards
enabling a high quality and safe healthcare system. Normalized data elements and clinical models can
help to ensure data consistency and comparability among users and systems, and formalized data
specifications can support the sharing of clinical knowledge among diverse jurisdictions with higher
efficiency and lower cost.
The goal of shareable, executable clinical guidelines is both worthwhile and challenging. One of the
greatest hurdles is that of representing the clinical information in a precise and shareable manner. The
Detailed Clinical Model (DCM) is an information model that is an essential part of the infrastructure
supporting the codification and standardization of evidence-based knowledge elements across
EHR systems. Each DCM is a logical model that shows the meaning, structure, possible content and
relationships between data elements for a given subject of EHR documentation. The coded elements
have explicit bindings to allowed coded values. Models are independent of any specific programming
language or type of database. DCMs support explicit, unambiguous query statements against data
instances. The DCM aids decision making by medical staff, and supports the valid reuse of medical data
for example for epidemiology, quality assurance and managerial goals.
The DCM for supporting evidence-based clinical practice is already in common use in the EHR systems of
some countries and regions. The Clinical Element Model of Intermountain Healthcare, templates defined
by HL7 International, archetypes defined by the openEHR Foundation and ISO 13606, the Clinical
Information Model in the Netherlands, and the Clinical Content Model by the Center for Interoperable
EHR in South Korea are examples of published DCM formalisms that can be used to organize the
clinical content of an EHR interoperability message and an EHR repository, share decision logic, and
build a data capture form of a clinical application. Each instance of a DCM defines how a corresponding
generic EHR representation is used to represent particular types of clinical information. Example DCMs
include representations for allergies, problem lists, laboratory tests, medication and diagnostic orders,
medication administration, adverse reaction, physical examination and clinical measurements, signs,
symptoms, diagnosis, clinical documents, procedure, family history, medical history and review of
symptoms.
According to the open EHR clinical community, medication, problem/diagnosis, adverse reaction,
vital signs group, laboratory report, alert, blood group, procedure, admission/episode and clinical
synopsis are the top ten priority archetype modelling areas. Examples of DCM instances might include
representations for documenting a pain symptom, heart sounds, liver function tests, a prescribed drug,
or a chest x-ray report. The standardization of EHR content through DCMs enables consistency and
interoperability in many possible application areas within healthcare, quality assurance, and research.
DCMs support multiple contexts as in the following:
— EHR data storage;
— message payload and service payload;
— exchanging clinical content;
— querying and analytics over clinical content;
— decision support over clinical content;
— entering clinical content;
— displaying clinical content;
— healthcare quality measures;
— clinical trials data (clinical research);
— normalization of data for secondary use;
— capture of coding output from NLP.
vi © ISO 2017 – All rights reserved

Whereas DCMs have the potential to improve clinical decision support and clinical documentation in
EHR systems, the critical challenge is to identify the qualitative and quantitative requirements of DCMs.
A few studies have suggested some quality requirements for DCMs (Table A.1). For the flexibility and
scalability of DCMs, general requirements for the system in which clinical data models are implemented
demand the following:
a) the addition of elements and attributes to the clinical model without the necessity of changing
the underlying software or database schema;
b) use of an existing formalism/syntax for the representation of the model;
c) binding of model attributes to standard terminology systems; and
d) the existence of a mechanism for stating ‘negation’.
General principles of good modelling include:
1) adoption of standard terminologies for use in the models;
2) representing the models in standard modelling languages;
3) sharing and approving the DCMs with a community of clinical experts;
4) defining data elements for decision support analysis of EHR clinical content.
DCM quality criteria have also been proposed where the following qualities were identified as being
important requirements of a good DCM: usefulness, desirability, the degree of use/acceptance in clinical
services, reusability, the degree of clinician introduction/validation, the proper use of vocabulary,
easy mapping to information models, applicability, application to other technologies, and ease of
maintenance.
Principles for the development of DCMs can be classified as principles pertaining to the structure of
the DCM, principles for creating the DCM content, and principles for the DCM development process. The
principles that pertain to the structure of DCMs contains information about the language formalism,
description of binding of attributes to standard terminologies, a strategy for supporting semantic links
among DCM instances, the definition of standard data types, and the description of standard units of
measure. The principles for DCM content creation emphasize the granularity, reusability, correctness,
and comprehensiveness of the models. Principles for the DCM development process emphasize
evidence-based model development, the need for proper use cases, use of metadata to track changes,
and compliance to the syntax of the modelling language.
There is common knowledge and understanding about the creation and utilization of DCMs. The
Clinical Information Modeling Initiative (CIMI) is an international collaboration that is dedicated to
providing a common format for DCMs so that semantically interoperable information may be created
and shared in health records, messages and documents. The strategic goal of CIMI is to enable sharing
of data, applications, reports, alerts, protocols, and decision support modules with anyone in the world.
“Plug-n-play” interoperability is the primary goal of CIMI. To accomplish these goals, CIMI is now
developing a shared repository of detailed clinical information models. CIMI is committed to making
these specifications openly available in a number of formats, beginning with the Archetype Definition
Language (ADL) from the openEHR Foundation (ISO 13606-2) and the Unified Modeling Language (UML)
from the Object Management Group (OMG), with the intent that the users of these specifications can
convert the models into their local formats. These formalisms are based on a common set of base data
types with formal bindings of the models to standard coded terminologies. The CIMI repository is open
to everyone and models are free for use at no cost.
The CIMI Reference Model Taskforce has established a set of requirements for the CIMI Reference
Model (RM) and Clinical Modeling Language respectively. The CIMI RM is the underlying Reference
Model on which CIMI’s clinical models are defined. The CIMI RM requirements define or recommend the
candidate elements that should be in the CIMI RM. The Clinical Modeling Taskforce is working to test the
candidate reference model in the development of a set of initial clinical information models. The Clinical
Modeling Language document lists the set of known requirements of the Clinical Modeling Language.
Many parties that have been involved in the various clinical modelling activities (13606/OpenEHR/HL7
templates/UML-DCM/CEML/Korean examples) are jointly working in CIMI.
Most of these aspects are specified in ISO/TS 13972, which provides characteristics, principles and
processes for development of detailed clinical models. With increasing research and implementation
of DCMs in international standards organizations and local health information systems as well, it is
important to ask how the models’ quality can be objectively measured. The last version of the CIMI RM
is 99,9 % consistent with ISO/TS 13972 characteristics. The remaining differences deal with archetype
specification typicalities which are not relevant to ISO/TS 13972, since the Technical Specification
covers multiple technical formalisms.
To illustrate the level of detail for a DCM, one example (Figure 1) is presented for heart rate, using the
ISO/TS 13972 format and the HL7 DCM project.
Figure 1 — DCM for heart rate example for the details required
Some quality requirements and criteria (see Table A.1) for DCMs proposed general principles of good
modelling, including:
i. adoption of standard terminologies for use in the models;
ii. representing the models in standard modelling languages;
iii. sharing and approving the DCMs with a community of clinical experts;
iv. defining models that can serve as unambiguous references for data used in decision modules;
v. allowing different styles of models that represent the same clinical data (though only one style
of model should be selected as the preferred style);
viii © ISO 2017 – All rights reserved

vi. applying the DCM Quality Management system (ISO/TS 13972) for DCM maintenance and
governance.
Most of these requirements also include metadata of DCM, design and development process, governance
and management as quality aspects of DCMs. Published requirements also specify process, product and
provenance-related DCM quality. However, one critical limitation of these attempts is that there is no
measurable metric to identify qualified DCMs for intended use.
If DCMs are to adequately support the EHR documentation needs of clinical practice, be endorsed by
clinical professional bodies and health services, and be adopted by vendors, these models have to be of
good quality, trusted, and in the future, certified. This document defines a set of quality metrics which
are required to evaluate the DCMs objectively. Quality metric for DCMs specifies:
— definition;
— evaluation target;
— evaluation method;
— evaluation result to evaluate DCMs.
This set of quality metrics, which constitutes essential qualitative and quantitative quality requirements
for DCMs, can be used to support rational decision-making by DCM developers and clinical users.
x © ISO 2017 – All rights reserved

TECHNICAL SPECIFICATION ISO/TS 18864:2017(E)
Health informatics — Quality metrics for detailed
clinical models
1 Scope
The purpose of this document is to define objective, reliable and reproducible quality metrics for
detailed clinical models (DCM).
This document specifies the principal metrics which are necessary and sufficient to evaluate DCMs.
The intended audiences of this document are:
— DCM developers, all users of information represented using DCMs, and evaluators of DCM quality;
— clinical and IT professionals of healthcare institutions;
— technical staff in the healthcare technology industry;
— experts involved in standards development;
— national and regional healthcare information technology leadership including certification bodies.
This document defines a set of quality metrics required to evaluate DCMs objectively. These quality
metrics can be used to support rational decision making by DCM developers who will have the essential
qualitative and quantitative quality requirements to use as guidelines as they create new content.
Clinical users can then use the quantitative assessments as they select models for specific use cases and
implement them in their clinical systems.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https://www.iso.org/obp
— IEC Electropedia: available at http://www.electropedia.org/
3.1
archetype
knowledge-related data structure that strongly supports semantic interoperability of EHRs
Note 1 to entry: They help to ensure reliable and easy sharing of meaningful sets of data between different
healthcare providers while allowing the re-use of their 'atomic' data components separately or within other
clinical models.
3.2
archetype model
information model of the metadata to represent the domain-specific characteristics of electronic health
record entries by specifying values or value constraints for classes and attributes in the electronic
health record reference model
[SOURCE: ISO 13606-1:—, 3.14]
3.3
archetype modeling language
formal language for expressing archetypes
Note 1 to entry: It provides a formal, textual syntax for describing constraints on a domain entity whose data are
described by an information model.
3.4
attribute
characteristic of an object or entity
Note 1 to entry: In the context of this standard: a specific characteristic of a data element.
[SOURCE: ISO/IEC 11179-1:2015, 3.1.1, modified]
3.5
concept definition
description of the attributes of a concept to delineate its meaning
3.6
conceptual model
description of common concepts and their relationships, particularly in order to facilitate exchange of
information between parties within a specific domain of healthcare
[SOURCE: ENV 1613 CR 12587, modified]
3.7
content quality
quality of DCM content as determined by a set of international standard-based or evidence-based
quality criteria
3.8
data
re-interpretable representation of information in a formalized manner suitable for communication,
interpretation or processing
[SOURCE: ISO/IEC 2382:2015, 2121272, modified]
3.9
data element
unit of data that is considered, in context, to be indivisible
[SOURCE: ISO/IEC 2382:2015, 2121599, modified]
3.10
data type
set of distinct values, characterized by properties of those values, and by operations on those values
[SOURCE: ISO/IEC 11404:2007, 3.12]
2 © ISO 2017 – All rights reserved

3.11
Detailed Clinical Model
DCM
relatively small, standalone information model designed to express a clinical concept in a standardized
and reusable manner
[SOURCE: ISO/TS 13972:2015, 2.22, modified]
3.12
logical Model
information model that specifies the structures and relationships between data elements but is
independent of any particular technology or implementation environment
3.13
electronic health record
EHR
logical representation of information regarding or relevant to the health of a subject of care
[SOURCE: ISO 18308:2011, 3.20, modified]
3.14
electronic health record architecture
EHRA
logical generic components from which electronic health records are designed, defined in terms of an
information model and computational services
[SOURCE: ISO 18308:2011, 3.21, modified]
3.15
entity
concrete or abstract thing of interest, including associations among things
[SOURCE: ISO/IEC 2382:2015, 2121433, modified]
3.16
entry
documentation of a discrete item of health information
Note 1 to entry: An entry may for example represent the documentation of a clinical observation, an inference, an
intention, a plan or an action.
[SOURCE: ISO 18308:2011, 3.24]
3.17
governance for Detailed Clinical Models
system by which development, distribution, and maintenance of detailed clinical models are directed
and controlled
[SOURCE: ISO/TS 13972:2015, 2.28, modified]
3.18
HL7 template
expression of a set of constraints on a Reference Information Model (RIM) or a RIM derived model that
is used to apply additional constraints to a portion of an instance of data which is expressed in terms of
some other Static Model, to further define and refine these existing models to specify a narrower and
more focused scope
Note 1 to entry: A template is represented by:
— a formal definition in one or more human readable languages or notations;
— [optionally] a formal definition as a static model;
— [optionally] one or more implementation specific representations that can be used to validate instances in a
particular context template.
[SOURCE: HL7 Ballot January 2010 Templates Project]
3.19
metadata
data that define and describe other data
[SOURCE: ISO/IEC 11179-1:2015, 3.2.16, modified]
3.20
metric
verifiable measure that captures performance in terms of how something is being done relative to a
standard, allows and encourages comparison, supports business strategy
3.21
model
representation of a domain that uses abstraction to express the relevant concepts
3.22
modelling
construction of abstract representations in the course of design, for example to represent the logical
structure of software applications before coding
Note 1 to entry: http://www.omg.org/gettingstarted/what_is_uml.htm
3.23
patient safety
prevention of harm caused by errors of commission and omission
[SOURCE: Reference 30]
3.24
quality
degree to which all the properties and characteristics of a product, process or service satisfy the
requirements which ensue from the purpose for which that product, process or service is to be used
[SOURCE: ISO 9000:2015, 3.6.2, modified]
3.25
quality management system
framework described by the ISO 9000 family of standards and comprised of three core elements:
quality control, quality assurance and quality improvement
3.26
quality measure
quantitative measure of one or more quality characteristics (two or more quality measures may be
required to specify one aspect of quality)
3.27
quality metric
agreed methods and means for measuring the levels of achievement, performance or conformance of a
component or its quality characteristics(s)
3.28
safety
freedom from unacceptable risk of harm
[SOURCE: ISO/IEC Guide 51:2014, 3.14, modified]
4 © ISO 2017 – All rights reserved

3.29
semantic interoperability
ability for data shared by systems to be understood at the level of fully defined domain concepts
[SOURCE: ISO 18308:2011, 3.45]
3.30
terminological system
ordered collection of concepts, in which each concept is expressed by terms, words or expressions
[SOURCE: ISO/IEC 11179-1:2015, 3.2.25, modified]
3.31
usage pattern
consistent and recurring use case for a particular detailed clinical model
3.32
value set
uniquely identifiable set of valid concept representations, where any concept representation can be
tested to determine whether or not it is a member of the value set
[SOURCE: HL7 Version 3 Core Principles]
Note 1 to entry: A value set is intended to be a set in the formal sense, and so should contain only one code for
each uniquely identifiable concept that it contains.
4 Abbreviated terms
For the purposes of this document, the following abbreviations apply.
EHR Electronic Health Record
DCM Detailed Clinical Model
CIMI Clinical Information Modeling Initiative
ADL Archetype Definition Language
AOM Archetype Object Model
CEM Clinical Element Model
CDA Clinical Document Architecture
CCM Clinical Contents Model
FHIR Fast Healthcare Interoperability Resources
UML Unified Modeling Language
OWL Web Ontology Language
XML Extensible Markup Language
OID Object Identifier
QFD Quality function deployment
5 Relationship with other activities
5.1 ISO/TS 13972
ISO/TS 13972 has been developed in order to leverage ISO 9000-based quality management in
the contexts of DCMs. The characteristics of DCMs, which are addressed in ISO/TS 13972, can be
transformed into measurable metrics as described in this document. Transformation of characteristics
to metrics can be one to many, many to one, or one to one.
5.2 ISO 13606-2
Archetype representation requirements published in ISO 13606-2 focused more on the technological
aspects of models. They are divided into requirements for archetype definition, archetype node
constraints, and data value constraints. They have been consulted, but not used directly, in developing
these quality metrics.
5.3 EuroRec quality criteria for archetypes
EuroRec, an organization that certifies EHRs in the European Union, led a research project funded by
the European Commission to develop criteria for the quality classification of EHR systems. The research
resulted in a set of archetype quality criteria, which covered administrative, clinical, technological,
information management, and repository operation requirements. Furthermore, they emphasized the
use of standard terms and modelling language, the construction of repositories for DCM sharing, and
the importance of metadata. Among the clinical requirements, the requirement for clinical use suggests
that the listing of accurate use patterns of clinical concepts, specification of whether the corresponding
archetype is used in a specific workflow, description of subject population groups, as well as expert
groups using an archetype, etc. are important to note.
5.4 Delphi study
Published criteria (see Table A.1) were used as the input criteria for a Delphi study that established
a core subset of criteria that were capable of more objective and formal assessment, published in
[12]
Reference. ISO/TS 13972, ISO 13606-2 and EuroRec quality have acted as the primary initial source
of the quality metrics presented in this document.
6 The purpose and importance of DCM quality
Why are we measuring the quality of DCMs? The role of DCMs is to minimize, and hopefully eliminate
much of the ambiguity in the clinical information shared across an eHealth infrastructure. Therefore,
the data represented in a DCM shall be accurate. The data shall accurately convey the concepts they
are intended to convey. To succeed in this role, DCMs shall, themselves, be fit for purpose. Good quality
clinical care relies on good quality information. A poor quality DCM is a potential exogenous source
of clinical error. To ensure quality, we employ both quality assurance and quality control. Quality
assurance is the adoption and measurable use of standardized processes intended to yield a good
quality work product; it is proactive. Quality control is the standardized measurement of the attributes
of a work product as compared to the established attributes of a good quality product; it is intended to
identify (and rectify) defects and is, therefore, reactive. Systemic quality control issues often uncover
underlying quality assurance issues and are therefore necessary for supporting kaizen (continuous
quality improvement).
See also ISO/TS 13972:2015, 5.1. This handles how DCM processes can be managed through a Quality
Management system (QMS). ISO/TS 13972 addresses the areas of analyses of clinical content and
context, involvement of clinicians and other stakeholders to verify DCM, acceptance, adoption and
usage of DCMs, use of a generic format for the conceptual and logical levels of DCM specifications for
their development, the deployment of a DCM QMS for maintenance, governance, and repositories, and
finally a DCM process monitoring and improvement approach. ISO/TS 13972 adapted this into a cyclical
process derived from ISO 9001/EN 15224.
6 © ISO 2017 – All rights reserved

7 Quality aspect of DCMs
According to some published quality criteria (see Table A.1) for the flexibility and scalability of DCMs,
general requirements for the system in which clinical data models are implemented demand the
following:
a) addition of elements and attributes to the clinical model without the necessity of changing the
underlying model, software or data base schema;
b) use of an existing formalism/syntax for the representation of the model;
c) binding of model attributes to standard terminology systems;
d) existence of a mechanism for stating ‘negation’. In this context, flexibility means that DCM designs
can adapt when external changes occur.
DCM metrics check to see if each DCM’s data elements and DCM metadata describe ‘required’ and
‘optional’ items. Scalability of DCMs refers to ability to enhance each DCM by adding new data elements
with minimal effort.
General principles of good modelling include:
1) adoption of standard terminologies for use in the models;
2) representing the models in standard modelling languages;
3) sharing and approving the DCMs with a community of clinical experts;
4) defining models that can serve as unambiguous references for data used in decision modules;
5) allowing different styles of models that represent the same clinical data (though only one style of
model should be selected as the preferred style);
6) applying the DCM Quality Management System (ISO/TS 13972) for DCM maintenance and
governance.
These published quality criteria describe the requirements to ensure the quality of DCMs. Most of these
requirements include metadata of DCM, design and development process, governance and management
as quality aspects of DCMs. Published requirements also specify process, product, and provenance-
related DCM quality. The published quality criteria focus on the appropriate expression of the content
and metadata.
8 DCM quality framework
8.1 DCM quality framework component
A consistent approach to representing DCM quality improvement activities will allow the integration
and indexing of target domains, characteristics, metrics, and expected benefits. This over-arching
framework identifies suitable metrics by which to assess and assure the quality aspects of DCMs.
For example, the purpose of ‘Per Data element’ target domain is to measure each DCM unit’s
comprehensiveness and coverage.
It is the intention that the quality framework will cover all quality aspects of DCMs:
— target domains;
— DCM quality characteristics;
— DCM quality metrics;
— expected benefits of high quality DCMs;
— use of the DCM quality management system for long term maintenance and governance of DCMs.
Figure 2 — Framework for the quality criteria for DCMs
8.2 DCM Quality characteristics
There are agreed attributes of a DCM by which its quality is assessed or measured. Most quality
characteristics are agreed quality aspects and component in ISO/TS 13972. Not all characteristics will
match evaluation target and measurement methods. Agreeing optimal metrics development approaches
shall be balanced with what is achievable within known resources.
8.3 DCM Quality metrics
— Definition
Definitions are offered to describe an evaluation target. It describes the evaluation area and
associated artifacts. All definitions consist of diverse depths regarding clinical, technical outcomes
and governing structure and process.
— Evaluation target
Evaluation target includes DCM metadata, DCM content, language, DCM processes, administrative
documents that are the subject of the evaluation.
— Evaluation method
Evaluation methods are of two types. In general, the primary users, such as clinicians and
researchers, review DCM contents and manually and subjectively determine whether they are fit
for purpose. Determining the correctness, specificity and the level of granularity is a clinical aspect
of the assessment of the contents of a DCM. Other metrics can be measured in an automatic and
objective way using technical functionalities.
— Evaluation result
The default evaluation result options presented in the metrics are divided into 4 categories:
8 © ISO 2017 – All rights reserved

1) meets the requirement (conforms);
2) does not meet the requirement (does not conform);
3) partially meets the requirement (partially conformant) and includes recommendation or
direction as to what is missing or deficient;
4) NA, NR (not relevant).
8.4 Expected benefits of high quality DCMs
— Patient safety
DCMs bring useful clinical contents to the delivery of healthcare, and it is making health care
systems safer and more efficient. There are specific patient safety risks that can be mitigated by
deploying standards-based DCMs within EHRs. However, patient safety risks can be introduced by
DCMs themselves unless there is a robust and relevant quality measure framework and related
quality measure activities. Good quality DCMs reduce ambiguity, and minimize any errors of
omission and errors of commission.
— Quality care
An important goal is that good information that supports high quality care is accurately stored
or conveyed in the EHR. The challenge that most modellers face is to appropriately structure the
clinical knowledge and data element specification without losing the meaning of clinical data in the
system. Evidence-based clinical guidelines and protocols shall be defined by reference to DCMs to
achieve high quality healthcare. Therefore, information which is properly defined in DCMs guides
and supports clinicians and other stakeholders in providing evidence-based good care. Based on
the metrics, the design, development, and management of DCM quality can be improved.
— Semantic interoperability
Representation of clinical semantics with standard structure supports patient safety and semantic
interoperability. Health service delivered using standards for the consistent and clear representation
of clinical data is essential. When quality of DCMs has been assured using the specified metrics, it
ensures that the meaning of information is preserved over time, between heterogeneous systems
and actors and care providers. Semantic interoperability is one of the key benefits of quality assured
DCMs. It enables the integration and safe use of computerized protocols and alerts and ensures the
meaningful and reliable use of clinical data for research and public health.
9 Quality metrics for DCMs
9.1 Design and development
9.1.1 General
There may be varying degrees of partial conformance to quality metrics, and the same holds true for
(full) conformance. It is thus deemed necessary to provide a range between greater or equal to (≥) 1
point, but less than (<) 2 points for partial conformance; and greater or equal to (≥) 2 points, but less
than (<) 4 points for (full) conformance. If a quality metric deals with a yes/no, or true/false question,
the score for the quality metric is 1 point for partial conformance, and 2 points for (full) conformance.
Otherwise, if partial or full conformance deals with an amount, number, or quality, the user should use
his/her own expert judgment in assigning a proper score, even a non-integer value for its score. It is,
however, reasonable to suggest that a value should be limited to the first digit after the decimal point.
Annex B provides an overview about typical detailed clinical modelling approaches.
...