ISO/IEC 19796-3:2009

INTERNATIONAL ISO/IEC
STANDARD 19796-3
First edition
2009-02-01
Information technology — Learning,
education and training — Quality
management, assurance and metrics —
Part 3:
Reference methods and metrics
Technologies de l'information — Apprentissage, éducation et
formation — Management, assurance et métrologie de la qualité —
Partie 3: Méthodes de référence et métrologie

Reference number
©
ISO/IEC 2009
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.

©  ISO/IEC 2009
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2009 – All rights reserved

Contents Page
Foreword .iv
Introduction.v
1 Scope.1
2 Conformance .2
3 Normative references.2
4 Terms and definitions .2
5 Mapping quality approaches.4
5.1 Quality approaches and ISO/IEC 19796 series.4
5.2 Practical guideline for quality approaches.6
6 Quality management / assurance activity model .6
7 Reference model for quality methods .8
7.1 Categories .8
7.2 Elements.10
8 Reference model for quality metrics .14
8.1 Categories .14
8.2 Elements.16
9 Collection of methods and metrics .19
9.1 Collection of methods.19
9.2 Collection of metrics .22
Annex A (informative) Description format examples of methods reference model.30
Annex B (informative) Description format examples of metrics reference model.34
Bibliography.38

© ISO/IEC 2009 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 19796-3 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 36, Information technology for learning, education and training.
ISO/IEC 19796 consists of the following parts, under the general title Information technology — Learning,
education and training — Quality management, assurance and metrics:
⎯ Part 1: General approach
⎯ Part 3: Reference methods and metrics
The following parts are under preparation:
⎯ Part 2: Harmonized quality model
⎯ Part 4: Best practice and implementation guide [Technical Report]
⎯ Part 5: How to use ISO/IEC 19796-1 [Technical Report]
iv © ISO/IEC 2009 – All rights reserved

Introduction
Quality in the field of distance education and e-learning has become an issue of increasing importance in
academia, institutions, and industry. A variety of approaches have been developed and implemented
successfully. Generic standards, such as ISO 9000:2005 and/or ISO 14000 have been used also in the
educational community. In addition to generic standards related to quality, there are specific quality guidelines
that have been developed and used for e-Learning or distance education (such as the ASTD criteria for e-
Learning, the BLA Quality Mark, Quality Platform Learning by D-ELAN, or Quality elements by Sloan-
consortium). It has become clear that quality management can contribute to improve the performance of
organizations in the field of learning, education, and training (LET).
Numerous approaches to quality management and assurance and their different scopes and objectives lead
to confusion within communities that depend on information technologies to support and facilitate learning,
education, and training. Therefore, a harmonized quality standard, the ISO/IEC 19796 series, has been
developed.
ISO/IEC 19796 has five parts as a series. ISO/IEC 19796-1 is the basic framework for quality development in
organizations within the field of learning, education, and training (LET). It serves as a framework to describe,
compare, and analyze quality management and quality assurance approaches. In addition to providing a
descriptive model that can be used to compare and analyze quality approaches, ISO/IEC 19796-1 identifies
the components of a seven-part process model within the lifecycle of information and communication systems
for learning, education, and training. As a framework it may be used to compare different quality assurance
and quality management processes. This part of ISO/IEC 19796 provides reference methods and metrics
used in the lifecycle process. It also provides examples of how methods and metrics can be generically
described, compared, and used for specific contexts. For each process in the e-learning lifecycle, a set of
potential methods and metrics should be specified. This set of methods and metrics can be used during the
development of an individual quality approach that is based on defined quality objectives.
ISO/IEC 19796-1 describes the processes for the e-Learning lifecycle. It is a reference model with a high level
of abstraction which has to be adapted to a certain organization and to a certain situation. To facilitate the
adaptation procedure, two reference models for quality management and quality assurance are described in
this part of ISO/IEC 19796:
⎯ reference model for methods;
⎯ reference model for metrics.
To facilitate the adaptation of ISO/IEC 19796-1, as the first step, this part of ISO/IEC 19796 provides
reference methods and metrics used in lifecycle processes with the reference models from ISO/IEC 19796-1.
It also provides collections of methods and metrics which are generically described and can be used for
specific contexts. For each process in the e-Learning lifecycle, a set of potential methods and metrics should
be specified. This set of methods and metrics can be used during the development and definition of an
individual quality approach based on certain quality objectives.
ISO/IEC 19796-1 provides a description format and a process framework as the reference framework for the
description of quality approaches (RFDQ), for the description and development of quality approaches. The
RFDQ framework is only a base for quality development – it does not provide specific methods and metrics for
particular quality objectives, and also does not provide instruments or procedures.
ISO Guide 72 Guidelines mainly distinguish three types of management system standards; Type A —
management systems requirements standards, Type B — management systems guidelines standards, and
Type C — management systems related standards. The consensus is to use ISO Guide 72 Guidelines for the
justification and development of management system standards. Accordingly, this International Standard is
Type C, a management systems related standard, as neither requirements are defined nor guidelines are
given.
© ISO/IEC 2009 – All rights reserved v

INTERNATIONAL STANDARD ISO/IEC 19796-3:2009(E)

Information technology — Learning, education and training —
Quality management, assurance and metrics —
Part 3:
Reference methods and metrics
1 Scope
This part of ISO/IEC 19796 extends the “reference framework for the description of quality approaches”
(RFDQ) defined in ISO/IEC 19796-1 by providing a harmonized description of the methods and metrics
required to implement quality management and quality assurance systems for stakeholders designing,
developing, or utilizing information technology systems used for learning, education, and training.
Activities such as quality planning, quality control, and quality improvement are important for quality
management implementations. While these three activities are focused on products, processes and their
development, quality assurance is focused more on confirmation and indication for internal and external
stakeholders. It should be noted that the reference methods and metrics for this part of ISO/IEC 19796 include
issues related to the implementation of quality management and assurance systems for information
technologies that are used for learning, education, and training. This part of ISO/IEC 19796 is to be used to
help identify methods and metrics to implement a quality assurance and management system of an IT system
used for learning, education, and training. For example, it may be used for quality management systems that
help to verify items such as IT system effectiveness, compliance with quality objectives including purposes,
customer satisfaction, training in the use of the IT system, complaints handling, and auditing.
NOTE While not included in the normative references of this part of ISO/IEC 19796 it may be helpful for ITLET
(Information Technology for Learning, Education, and Training) stakeholders interested in quality issues to refer to the
related standards listed in the bibliography for further information regarding quality management and quality assurance.
During the implementation of quality management and quality assurance systems, using specific methods and
metrics are indispensable for the exchange, purchase, management, and archiving of learning courses,
systems, and/or services. Involving all stakeholders (e.g., developers, administrators, government, providers,
teachers, and learners) within a framework that is open and inclusive will help to ensure that information
technologies for learning, education, and training are both effective for and appropriate to learning and
teaching needs.
This part of ISO/IEC 19796 has the following components, which are developed in order to indicate and
communicate quality approaches.
• The reference model for methods — provides an interoperable formalized description of methods that
can be easily implemented and adopted into a quality management system to realize and facilitate
quality management.
• The reference model for metrics — provides an interoperable formalized descri
...