ISO 19114:2003
(Main)Geographic information - Quality evaluation procedures
Geographic information - Quality evaluation procedures
ISO 19114:2003 provides a framework of procedures for determining and evaluating quality that is applicable to digital geographic datasets, consistent with the data quality principles defined in ISO 19113. It also establishes a framework for evaluating and reporting data quality results, either as part of data quality metadata only, or also as a quality evaluation report. ISO 19114:2003 is applicable to data producers when providing quality information on how well a dataset conforms to the product specification, and to data users attempting to determine whether or not the dataset contains data of sufficient quality to be fit for use in their particular applications. Although ISO 19114:2003 is applicable to all types of digital geographic data, its principles can be extended to many other forms of geographic data such as maps, charts and textual documents.
Information géographique — Procédures d'évaluation de la qualité
Geografske informacije - Postopki za ocenjevanje kakovosti
General Information
Relations
Frequently Asked Questions
ISO 19114:2003 is a standard published by the International Organization for Standardization (ISO). Its full title is "Geographic information - Quality evaluation procedures". This standard covers: ISO 19114:2003 provides a framework of procedures for determining and evaluating quality that is applicable to digital geographic datasets, consistent with the data quality principles defined in ISO 19113. It also establishes a framework for evaluating and reporting data quality results, either as part of data quality metadata only, or also as a quality evaluation report. ISO 19114:2003 is applicable to data producers when providing quality information on how well a dataset conforms to the product specification, and to data users attempting to determine whether or not the dataset contains data of sufficient quality to be fit for use in their particular applications. Although ISO 19114:2003 is applicable to all types of digital geographic data, its principles can be extended to many other forms of geographic data such as maps, charts and textual documents.
ISO 19114:2003 provides a framework of procedures for determining and evaluating quality that is applicable to digital geographic datasets, consistent with the data quality principles defined in ISO 19113. It also establishes a framework for evaluating and reporting data quality results, either as part of data quality metadata only, or also as a quality evaluation report. ISO 19114:2003 is applicable to data producers when providing quality information on how well a dataset conforms to the product specification, and to data users attempting to determine whether or not the dataset contains data of sufficient quality to be fit for use in their particular applications. Although ISO 19114:2003 is applicable to all types of digital geographic data, its principles can be extended to many other forms of geographic data such as maps, charts and textual documents.
ISO 19114:2003 is classified under the following ICS (International Classification for Standards) categories: 35.240.70 - IT applications in science. The ICS classification helps identify the subject area and facilitates finding related standards.
ISO 19114:2003 has the following relationships with other standards: It is inter standard links to ISO 19157:2013. Understanding these relationships helps ensure you are using the most current and applicable version of the standard.
You can purchase ISO 19114:2003 directly from iTeh Standards. The document is available in PDF format and is delivered instantly after payment. Add the standard to your cart and complete the secure checkout process. iTeh Standards is an authorized distributor of ISO standards.
Standards Content (Sample)
INTERNATIONAL ISO
STANDARD 19114
First edition
2003-08-15
Geographic information — Quality
evaluation procedures
Information géographique — Procédures d'évaluation de la qualité
Reference number
©
ISO 2003
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO 2003
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO 2003 — All rights reserved
Contents Page
Foreword. v
Introduction . vi
1 Scope. 1
2 Conformance . 1
3 Normative references . 1
4 Terms and definitions. 1
5 Abbreviated terms. 2
6 Process for evaluating data quality . 3
6.1 General. 3
6.2 Components of the process. 3
7 Data quality evaluation methods. 4
7.1 Classification of data quality evaluation methods . 4
7.2 Direct evaluation methods . 5
7.3 Indirect evaluation method . 6
7.4 Data quality evaluation examples . 7
8 Reporting data quality evaluation information . 7
8.1 Reporting as metadata . 7
8.2 Reporting in a quality evaluation report . 7
8.3 Reporting aggregated data quality result. 7
Annex A (normative) Abstract test suites. 8
A.1 Introduction . 8
A.2 Quality evaluation procedures . 8
A.3 Evaluating data quality. 8
A.4 Reporting data quality . 8
Annex B (informative) Uses of quality evaluation procedures . 9
B.1 Introduction . 9
B.2 Development of a product specification or user requirements . 9
B.3 Quality control during dataset creation. 9
B.4 Inspection for conformance to a product specification. 9
B.5 Evaluation of dataset conformance to user requirements . 9
B.6 Quality control during dataset update . 9
Annex C (informative) Applying quality evaluation procedures to dynamic datasets. 10
C.1 Introduction . 10
C.2 Determining and reporting the quality of a dynamic dataset. 10
C.3 Establishing continuous quality evaluation procedures . 10
C.4 Periodically re-establish the reference quality of the dataset. 11
Annex D (informative) Examples of data quality measures . 12
D.1 Introduction . 12
D.2 Relationship of the data quality components . 12
D.3 Examples of data quality completeness measures. 14
D.4 Examples of data quality logical consistency measures. 15
D.5 Examples of data quality positional accuracy measures . 19
D.6 Examples of data quality temporal accuracy measures . 23
D.7 Examples of data quality thematic accuracy measures . 26
Annex E (informative) Guidelines for sampling methods applied to geographic datasets . 30
E.1 Introduction.30
E.2 Lot and item .30
E.3 Sample size .30
E.4 Sampling strategies .31
E.5 Probability-based sampling .34
Annex F (informative) Example of testing for thematic accuracy and completeness .36
F.1 Introduction.36
F.2 Quality evaluation process.36
F.3 Method for data quality evaluation.36
F.4 Inspection for quality .37
F.5 Determination of data quality results and conformance.38
F.6 Reporting quality results .39
Annex G (informative) Example of measurement and reporting of completeness and thematic
accuracy .42
G.1 Introduction.42
G.2 Dataset description .42
G.3 Evaluation of data quality.47
G.4 Reporting quality results .50
Annex H (informative) Example of an aggregated data quality result.53
H.1 Introduction.53
H.2 Dataset description .53
H.3 Universe of discourse.54
H.4 Dataset.55
H.5 Aggregation of evaluation results and reporting.55
Annex I (normative) Reporting quality information in a quality evaluation report .57
I.1 Introduction.57
I.2 Quality evaluation report components.57
Annex J (informative) Aggregation of data quality results.61
J.1 Introduction.61
J.2 100 % pass/fail.61
J.3 Weighted pass/fail.61
J.4 Subset of results sufficient for product purpose.62
J.5 Maximum/minimum value.62
Bibliography.63
iv © ISO 2003 — All rights reserved
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies
(ISO member bodies). The work of preparing International Standards is normally carried out through ISO
technical committees. Each member body interested in a subject for which a technical committee has been
established has the right to be represented on that committee. International organizations, governmental and
non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the
International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of technical committees is to prepare International Standards. Draft International Standards
adopted by the technical committees are circulated to the member bodies for voting. Publication as an
International Standard requires approval by at least 75 % of the member bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO shall not be held responsible for identifying any or all such patent rights.
ISO 19114 was prepared by Technical Committee ISO/TC 211, Geographic information/Geomatics.
Introduction
For the purpose of evaluating the quality of a dataset, clearly defined procedures must be used in a consistent
manner. This enables data producers to express how well their product meets the criteria set forth in its
product specification and enables data users to establish the extent to which a dataset meets their
requirements. The quality of a dataset is described using two components: a quantitative component and a
non-quantitative component. The objective of this International Standard is to provide guidelines for evaluation
procedures of quantitative quality information for geographic data in accordance with the quality principles
described in ISO 19113. It also offers guidance on reporting quality information.
This International Standard recognizes that a data producer and a data user may view data quality from
different perspectives. Conformance quality levels can be set using the data producer’s product specification
or a data user’s data quality requirements. If the data user requires more data quality information than that
provided by the data producer, the data user may follow the data producer’s data quality evaluation process
flow to get the additional information. In this case, the data user requirements are treated as a product
specification for the purpose of using the data producer process flow.
The quality evaluation procedures described in this International Standard, when applied in accordance with
ISO 19113, provide a consistent and standard manner to determine and report the quality information in a
dataset.
vi © ISO 2003 — All rights reserved
INTERNATIONAL STANDARD ISO 19114:2003(E)
Geographic information — Quality evaluation procedures
1 Scope
This International Standard provides a framework of procedures for determining and evaluating quality that is
applicable to digital geographic datasets, consistent with the data quality principles defined in ISO 19113. It
also establishes a framework for evaluating and reporting data quality results, either as part of data quality
metadata only, or also as a quality evaluation report.
This International Standard is applicable to data producers when providing quality information on how well a
dataset conforms to the product specification, and to data users attempting to determine whether or not the
dataset contains data of sufficient quality to be fit for use in their particular applications.
Although this International Standard is applicable to all types of digital geographic data, its principles can be
extended to many other forms of geographic data such as maps, charts and textual documents.
2 Conformance
This International Standard defines three classes of conformance: one for quality evaluation procedures, one
for evaluating data quality, and one for reporting quality information. The abstract test suites for the three
classes of conformance are given in Annex A.
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
ISO 19113:2002, Geographic information — Quality principles
ISO 19115:2003, Geographic information — Metadata
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 19113 and ISO 19115 (some of
which are repeated for convenience) and the following apply.
4.1
conformance quality level
threshold value or set of threshold values for data quality results used to determine how well a dataset meets
the criteria set forth in its product specification or user requirements
4.2
dataset
identifiable collection of data
[ISO 19115]
NOTE A dataset may be a smaller grouping of data which, though limited by some constraint such as spatial extent
or feature type, is located physically within a larger dataset. For purposes of data quality evaluation, a dataset may be as
small as a single feature or feature attribute contained within a larger dataset.
4.3
dataset series
collection of datasets sharing the same product specification
[ISO 19115]
4.4
direct evaluation method
method of evaluating the quality of a dataset based on inspection of the items within the dataset
4.5
full inspection
inspection of every item in a dataset
NOTE Full inspection is also known as 100 % inspection.
4.6
indirect evaluation method
method of evaluating the quality of a dataset based on external knowledge
NOTE Examples of external knowledge are dataset lineage, such as production method or source data.
4.7
item
that which can be individually described or considered
[ISO 2859-1]
NOTE An item can be any part of a dataset, such as a feature, feature relationship, feature attribute, or combination
of these.
4.8
population
totality of items under consideration
[ISO 3534-2]
EXAMPLE 1 All points in a dataset.
EXAMPLE 2 Names of all roads in a certain geographic area.
4.9
reference data
data accepted as representing the universe of discourse, to be used as reference for direct external quality
evaluation methods
5 Abbreviated terms
ADQR aggregated data quality results
AQL acceptance quality limit [ISO 3534-2]
RMSE root mean square error
2 © ISO 2003 — All rights reserved
6 Process for evaluating data quality
6.1 General
A quality evaluation process may be used in different phases of a product life cycle, having different objectives
in each phase. The phases of the life cycle considered here are specification, production, delivery, use and
update. Annex B describes some specific dataset-related operations to which quality evaluation procedures
are applicable.
The process for evaluating data quality is a sequence of steps to produce and report a data quality result. A
quality evaluation process consists of the application of quality evaluation procedures to specific dataset-
related operations performed by the dataset producer and the dataset user.
Processes for evaluating data quality are applicable to static datasets and to dynamic datasets. Dynamic
datasets are datasets that receive updates so frequently that for all practical purposes they are continuously
changing. Annex C describes the application of the process to evaluate data quality to dynamic datasets.
6.2 Components of the process
6.2.1 Process flow
The quality evaluation process is a sequence of steps taken to produce a quality evaluation result. Figure 1
illustrates the process flow for evaluating and reporting data quality results.
Figure 1 — Evaluating and reporting data quality results
6.2.2 Process steps
Table 1 specifies the process steps.
Table 1 — Process steps
Process Action Description
step
1 Identify an applicable data quality The data quality element, data quality sub-element, and data quality
element, data quality sub-element, scope to be tested is identified in accordance with the requirements
and data quality scope of ISO 19113. This is repeated for as many different tests as required
by the product specification or user requirements.
2 Identify a data quality measure A data quality measure, data quality value type and, if applicable, a
data quality value unit is identified for each test to be performed.
Annex D provides examples of data quality measures for the data
quality elements and data quality sub-elements given in ISO 19113.
Annex D, by these examples, provides assistance to the user in
selection of a measure.
3 Select and apply a data quality A data quality evaluation method for each identified data quality
evaluation method measure is selected.
NOTE A spatial description of the results (achievable by spatial
interpolation of the results, graphical portrayal, etc.) might be useful,
corresponding not to a result, but to a different, although related, dataset.
4 Determine the data quality result A quantitative data quality result, a data quality value or set of data
quality values, a data quality value unit and a date are the output of
applying the method.
5 Determine conformance Whenever a conformance quality level has been specified in the
product specification or user requirements, the data quality result is
compared with it to determine conformance. A conformance data
quality result (pass-fail) is the comparison of the quantitative data
quality result with a conformance quality level.
7 Data quality evaluation methods
7.1 Classification of data quality evaluation methods
A data quality evaluation procedure is accomplished through the application of one or more data quality
evaluation methods. Data quality evaluation methods are divided into two main classes: direct and indirect.
Direct methods determine data quality through the comparison of the data with internal and/or external
reference information. Indirect methods infer or estimate data quality using information on the data, such as
lineage. The direct evaluation methods are further subclassified by the source of the information needed to
perform the evaluation. Figure 2 depicts this classification structure.
Figure 2 — Classification of data quality evaluation methods (informative)
4 © ISO 2003 — All rights reserved
7.2 Direct evaluation methods
7.2.1 Types of direct evaluation methods
The direct evaluation method is further subdivided into internal and external. All the data needed to perform an
internal direct data quality evaluation method are internal to the dataset being evaluated.
EXAMPLE 1 All the data necessary to perform a logical consistency test for topological consistency of boundary
closure resides in a topologically structured dataset.
External direct quality evaluation requires reference data external to the dataset being tested.
EXAMPLE 2 The data needed to perform a completeness test for the road names in a dataset requires another
information source of road names.
EXAMPLE 3 A positional accuracy test requires a reference dataset or a new survey.
7.2.2 Means of accomplishing direct evaluation
For both external and internal evaluation methods, there are two considerations, automated or non-automated
and full inspection or sampling.
Data quality elements and data quality sub-elements which are easily checked by automated means include
the following:
a) logical consistency: format consistency;
EXAMPLE Check data fields for positive entry.
b) logical consistency: topological consistency;
EXAMPLE Polygon closure.
c) logical consistency: domain consistency;
EXAMPLE Bounds violations, specified domain value violations.
d) completeness: omission;
EXAMPLE Comparison check of street names from another file.
e) completeness: commission;
EXAMPLE Comparison check of street names from another file.
f) temporal accuracy: temporal consistency.
EXAMPLE Check all records for appropriate range of dates.
7.2.3 Full inspection
Full inspection requires testing every item in the population specified by the data quality scope. Table 2
describes the procedure for full inspection that shall be used.
Table 2 — Procedure for full inspection
Procedure step Description
Define items An item is a minimum unit to be inspected. An item can be a feature, a
feature attribute or a feature relationship.
Inspect items in the data quality scope Inspect every item it the data quality scope.
NOTE Full inspection is most appropriate for small populations or for tests that can be accomplished by automated
means.
7.2.4 Sampling
Sampling requires testing sufficient items in the population in order to achieve a data quality result. Table 3
describes the sampling procedure that shall be used.
Table 3 — Sampling procedure
Procedure step Description
Define a sampling method Examples of sampling methods are given in Annex E. Those methods
include simple random sampling, stratified sampling (e.g. guided by
feature type, a feature relationship or an area), multistage sampling and
non-random sampling.
Define items An item is a minimum unit to be inspected. An item can be a feature, a
feature attribute or a feature relationship.
Divide data quality scope (population) into lots A lot is a collection of items in the data quality scope from which a
sample is drawn and inspected. Each lot should, as far as possible,
consist of items produced under the same conditions and at the same
time.
Divide lots to sampling units Sampling unit is the area of the lot where inspection is conducted.
Define the sampling ratio or sample size A sampling ratio gives information on how many items on average are
extracted for inspection from each lot.
Select sampling units Select required number of sampling units so that the sampling ratio or
sample size for items is fulfilled.
Inspect items in the sampling units Inspect every item in the sampling units.
The sampling procedure shall be reported in accordance with Clause 8.
The ISO 2859 series and ISO 3951-1 may be applied to sampling for evaluating conformance to a product
specification. These standards were originally developed for non-spatial use. Annex E of this International
Standard gives examples describing how to apply the ISO 2859 series and ISO 3951-1 and provides
guidelines on how to define samples and devise sampling methods, taking the geographic nature of the data
into account.
The reliability of the data quality result should be analysed when using sampling, especially when using small
sample sizes and methods other than simple random sampling.
7.3 Indirect evaluation method
The indirect evaluation method is a method of evaluating the quality of a dataset based on external knowledge.
This external knowledge may include, but is not limited to, data quality overview elements and other quality
reports on the dataset or data used to produce the dataset.
NOTE 1 This method is recommended only if direct evaluation methods cannot be used.
6 © ISO 2003 — All rights reserved
NOTE 2 Usage information records uses of a dataset. This is helpful when searching for datasets that have been
produced or used for specific purposes.
NOTE 3 Lineage information records information about the production and history of the dataset. It includes information
about, for example, source materials to produce a dataset or the production processes applied. This is useful when
determining the suitability of a dataset for a given use. An example is lineage metadata relating to a digital terrain model
file that has been created by means of stereo-correlation from images captured under certain conditions. Experience tells
the evaluator that the horizontal positional RMSE is 10 m for this type of imagery. Or, for example, lineage metadata of a
digitized 1:25 000 scale topographic map indicates conformance to a town planner’s requirements for a base map.
NOTE 4 Purpose information describes the purpose for which the dataset was produced. A purpose may be in support
of a specific requirement, or the dataset may be a general purpose dataset for several uses. This is useful when
identifying the possible value of a dataset.
7.4 Data quality evaluation examples
Examples of typical methods used and how they may be applied are described in Annexes F, G and H.
8 Reporting data quality evaluation information
8.1 Reporting as metadata
Quantitative quality results shall be reported as metadata in compliance with ISO 19115, which contains the
related model and data dictionary.
8.2 Reporting in a quality evaluation report
There are two conditions under which a quality evaluation report shall be produced:
a) when data quality results reported as metadata are only reported as pass/fail;
b) when aggregated data quality results are generated.
The report is required in the latter condition to explain how aggregation was done and how to interpret the
meaning of the aggregate result. However, a quality evaluation report may be created at any other time (such
as to provide more detail than reported as metadata) but a quality evaluation report cannot be used in lieu of
reporting as metadata.
A quality evaluation report shall be produced in compliance with Annex I which contains the relevant model
and data dictionary.
8.3 Reporting aggregated data quality result
When several quality results are aggregated into a single quality result for reporting the quality of a dataset,
the aggregated data quality result shall be reported as metadata and shall be included in the data quality
report. The data quality result shall be reported as type “aggregate”. Annex J describes the production of
aggregate data quality results and Annex H provides a production example.
Annex A
(normative)
Abstract test suites
A.1 Introduction
This annex defines three classes of conformance
quality evaluation procedure (A.2),
evaluating data quality (A.3), and
reporting data quality (A.4).
Any quality evaluation procedures claiming conformance with this International Standard shall pass all the
requirements given in A.2. Any evaluation of data quality claiming conformance with this International
Standard shall pass all the requirements given in A.3. Any report of data quality claiming conformance with
this International Standard shall pass all the requirements given in A.4.
NOTE All of the test cases are of test type “basic”.
A.2 Quality evaluation procedures
Abstract test suite for class 1 shall be as follows.
a) Test purpose: to assure the quality evaluation procedure was produced in accordance with this
International Standard.
b) Test method: pass all the requirements described in A.3 and A.4.
c) Reference: A.3 and A.4.
A.3 Evaluating data quality
Abstract test suite for class 2 shall be as follows.
a) Test purpose: to assure the quality evaluation procedure was produced in accordance with the quality
evaluation process in Clause 6.
b) Test method: compare the quality evaluation procedure with the quality evaluation as appropriate.
c) Reference: ISO 19114:2003, Clause 6.
A.4 Reporting data quality
Abstract test suite for class 3 shall be as follows.
a) Test purpose: to assure data quality has been reported in accordance with Clause 8.
b) Test method: compare the quality evaluation reported to assure data quality results were appropriately
reported in accordance with Clause 8 and applicable annexes.
c) Reference: ISO 19114:2003, Clause 8.
8 © ISO 2003 — All rights reserved
Annex B
(informative)
Uses of quality evaluation procedures
B.1 Introduction
Quality evaluation procedures may be used in different phases of a product life cycle. This annex provides
examples of stages of a product's life cycle during which quality evaluation procedures may be applied.
B.2 Development of a product specification or user requirements
When developing a product specification or user requirement, quality evaluation procedures may be used to
assist in establishing conformance quality levels that should be met by the final product. A product
specification or user requirement should include conformance quality levels for the dataset and quality
evaluation procedures to be applied during production and updating.
B.3 Quality control during dataset creation
At the production stage, the producer may apply quality evaluation procedures, either explicitly established or
not contained in the product specification, as part of the process of quality control. The description of the
applied quality evaluation procedures, when used for production quality control, should be reported as lineage
metadata including, but not necessarily limited to, the quality evaluation procedures applied, conformance
quality levels established and the results.
B.4 Inspection for conformance to a product specification
On completion of the production, a quality evaluation process is used to produce and report data quality
results. These results may be used to determine whether a dataset conforms to its product specification. If the
dataset passes inspection (composed of a set of quality evaluation procedures), the dataset is considered to
be ready for use. The results of the inspection operation should be reported in accordance with Clause 8.
The outcome of the inspection will be either acceptance or rejection of the dataset. If the dataset is rejected,
then after the data have been corrected, a new inspection will be required before the product can be deemed
to be in conformance with the product specification.
B.5 Evaluation of dataset conformance to user requirements
Quality evaluation procedures are used to establish the conformance quality levels for a dataset to meet a
user requirement. Indirect as well as direct methods may be used in analyses of dataset conformance to user
requirements. The results of the quality evaluation for conformance to user requirements may be reported as
usage metadata for the dataset.
B.6 Quality control during dataset update
Quality evaluation procedures are applied to dataset update operations, both to the items being used for
update and to benchmark the quality of the dataset after update has occurred. Guidance for the use of
ISO 19113 and this International Standard on dynamic datasets is given in Annex C.
Annex C
(informative)
Applying quality evaluation procedures to dynamic datasets
C.1 Introduction
This annex describes how quality evaluation procedures may be applied to dynamic datasets. Here dynamic
datasets are defined as datasets that receive updates so frequently that, for all practical purposes, they are
continuously being updated. For example, an online cadastre dataset may receive updates every few minutes.
There are basically two ways to determine and report the quality of a dynamic dataset: benchmark and
continuous.
C.2 Determining and reporting the quality of a dynamic dataset
C.2.1 Benchmark procedure
The benchmark procedure is based on the establishment of a suitable reporting frequency (e.g. weekly or tri-
monthly) and making a copy of the dataset at the reporting date. Then the copy is tested as if it were a static
dataset. This type of testing and reporting will provide the quality of the dataset as of the date/time of the copy.
C.2.2 Continuous procedure
The continuous procedure is based on testing the updates and evaluating the impact of the updates. This is
equivalent to embedding the quality evaluation procedures given in this International Standard into an
process-oriented procedure (e.g. that given in ISO 9001). Since this procedure only can provide the current
status of the quality of the updated items, it is necessary to combine both benchmark and continuous
procedures as described in C.3 in order to establish the quality of the updated database.
C.3 Establishing continuous quality evaluation procedures
C.3.1 Identify the parts
In accordance with the steps given in 6.2, identify applicable data quality elements and their associated data
quality sub-elements, data quality scopes, data quality measure, and conformance quality levels to be used in
the evaluation and reporting of the results.
C.3.2 Select the method to be applied
Select the data quality evaluation method to be applied. Then the evaluation will be on the updated feature
and the relationship of that feature with the others within the data quality scope. In continuous quality
evaluation procedures, only indirect or internal direct methods may be applied.
EXAMPLES
a) Is the update from a trusted source?
b) Does the update preserve topological consistency?
c) Does the address of the feature updated retain logical consistency?
10 © ISO 2003 — All rights reserved
C.3.3 Establish a dataset quality reference
Use the benchmark procedure to establish reference values for the quality of the dataset for the features and
feature attributes within scope to be checked during the continuous testing.
C.3.4 Integrate continuous tests into update process
Integrate the continuous tests into the update process flow so that each proposed update is tested and
accepted before it is introduced into the dataset.
C.3.5 Dynamically update data quality results
By integrating the continuous tests into the update process flow, each accepted update causes the current
quality results to be adjusted accordingly. This will allow for immediate reports on dataset quality to be
generated.
C.4 Periodically re-establish the reference quality of the dataset
All aspects of the quality of a dataset may not be tested through a continuous process-based operation. For
example, omission of features may not be found when only updated items are tested. The dataset should be
subject to periodic benchmark-type quality procedures.
Annex D
(informative)
Examples of data quality measures
D.1 Introduction
This annex provides simple examples of data quality measures for each data quality element and its
associated sub-elements, defined in ISO 19113, to demonstrate how the data quality components relate
during an evaluation operation. More detailed examples may be found in other annexes of this International
Standard.
For each data quality element and sub-element combination, an example data quality scope is given together
with example dataset parameters. Then three data quality measures are shown, each designed to
demonstrate a different way to evaluate quality. The examples are as complete as possible; a data quality
date and conformance quality level are given. Finally, an interpretation of the data quality result is given as an
example of the quality result meaning.
While the examples given in this annex are simple, it may be desirable to refer to them in profiles or other
documents. Therefore, this annex has a data quality measure identification code which relates the example to
the data quality element and data quality sub-element.
D.2 Relationship of the data quality components
Table D.1 gives the relationship of the data quality components.
In order to save space, each data quality component has been given a short name that will be used
throughout this annex.
12 © ISO 2003 — All rights reserved
Table D.1 — Relationship of data quality components
a
Data quality components Component domain Example
Short name
Data quality scope DQ_Scope Free text All items classified as houses
Data quality element DQ_Element Enumerated domain 1 – Completeness
1 – completeness data quality element
2 – logical consistency describing the presence or
3 – positional accuracy absence of features, their
4 – temporal accuracy attributes and their
5 – thematic accuracy relationships
Data quality subelement DQ_Subelement Enumerated domain 1 – Commission
(Dependent upon data quality excess data in the dataset
element)
EXAMPLE
Data quality measure DQ_Measure
Data quality measure DQ_MeasureDesc Free text Existence of excess items
description
Data quality measure DQ_MeasureID Enumerated domain 10101
identification code
Data quality evaluation method DQ_EvalMethod
Data quality evaluation DQ_EvalMethodType Enumerated domain 2 – external
method type 1 – internal (direct)
2 – external (direct)
3 – indirect
Data quality evaluation DQ_EvalMethodDesc Free text or citation (depends Compare count of items in
method description on data quality evaluation dataset against count of items
method type) in universe of discourse
Data quality result DQ_QualityResult
Data quality value type DQ_ValueType Enumerated domain 1 – Boolean variable
1 – Boolean variable
2 – number
3 – ratio
4 – percentage
5 – sample
6 – table
7 – binary image
8 – matrix
9 – citation (ISO 19115)
10 – free text
11 – other
Data quality value DQ_Value Record (ISO 11404) True
(Depends on data quality
value type )
Data quality value unit DQ_ValueUnit (Depends on data quality Not applicable
value)
Data quality date DQ_Date ISO 8601:1988 2000-03-05
Conformance quality level DQ_ConformanceLevel value or set of values Zero difference between
dataset and universe of
discourse counts
a
Short name is for use within this annex.
D.3 Examples of data quality completeness measures
Completeness is the presence or absence of features, their attributes and their relationships. It has the
following sub-elements:
commission: excess data in a dataset;
omission: data absent from a dataset.
Table D.2 provides some examples for the sub-elements.
Table D.2 — Examples of data quality completeness measures
Data quality component Example 1 Example 2 Example 3
DQ_Scope All items classified as houses All items classified as houses All items classified as houses
in the dataset. and bounded by longitudes and in the town of Augusta,
−83,1 −83,3 and lat
...
SLOVENSKI STANDARD
01-december-2003
Geografske informacije - Postopki za ocenjevanje kakovosti
Geographic information -- Quality evaluation procedures
Information géographique -- Procédures d'évaluation de la qualité
Ta slovenski standard je istoveten z: ISO 19114:2003
ICS:
03.120.99 Drugi standardi v zvezi s Other standards related to
kakovostjo quality
07.040 Astronomija. Geodezija. Astronomy. Geodesy.
Geografija Geography
35.240.70 Uporabniške rešitve IT v IT applications in science
znanosti
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
INTERNATIONAL ISO
STANDARD 19114
First edition
2003-08-15
Geographic information — Quality
evaluation procedures
Information géographique — Procédures d'évaluation de la qualité
Reference number
©
ISO 2003
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO 2003
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO 2003 — All rights reserved
Contents Page
Foreword. v
Introduction . vi
1 Scope. 1
2 Conformance . 1
3 Normative references . 1
4 Terms and definitions. 1
5 Abbreviated terms. 2
6 Process for evaluating data quality . 3
6.1 General. 3
6.2 Components of the process. 3
7 Data quality evaluation methods. 4
7.1 Classification of data quality evaluation methods . 4
7.2 Direct evaluation methods . 5
7.3 Indirect evaluation method . 6
7.4 Data quality evaluation examples . 7
8 Reporting data quality evaluation information . 7
8.1 Reporting as metadata . 7
8.2 Reporting in a quality evaluation report . 7
8.3 Reporting aggregated data quality result. 7
Annex A (normative) Abstract test suites. 8
A.1 Introduction . 8
A.2 Quality evaluation procedures . 8
A.3 Evaluating data quality. 8
A.4 Reporting data quality . 8
Annex B (informative) Uses of quality evaluation procedures . 9
B.1 Introduction . 9
B.2 Development of a product specification or user requirements . 9
B.3 Quality control during dataset creation. 9
B.4 Inspection for conformance to a product specification. 9
B.5 Evaluation of dataset conformance to user requirements . 9
B.6 Quality control during dataset update . 9
Annex C (informative) Applying quality evaluation procedures to dynamic datasets. 10
C.1 Introduction . 10
C.2 Determining and reporting the quality of a dynamic dataset. 10
C.3 Establishing continuous quality evaluation procedures . 10
C.4 Periodically re-establish the reference quality of the dataset. 11
Annex D (informative) Examples of data quality measures . 12
D.1 Introduction . 12
D.2 Relationship of the data quality components . 12
D.3 Examples of data quality completeness measures. 14
D.4 Examples of data quality logical consistency measures. 15
D.5 Examples of data quality positional accuracy measures . 19
D.6 Examples of data quality temporal accuracy measures . 23
D.7 Examples of data quality thematic accuracy measures . 26
Annex E (informative) Guidelines for sampling methods applied to geographic datasets . 30
E.1 Introduction.30
E.2 Lot and item .30
E.3 Sample size .30
E.4 Sampling strategies .31
E.5 Probability-based sampling .34
Annex F (informative) Example of testing for thematic accuracy and completeness .36
F.1 Introduction.36
F.2 Quality evaluation process.36
F.3 Method for data quality evaluation.36
F.4 Inspection for quality .37
F.5 Determination of data quality results and conformance.38
F.6 Reporting quality results .39
Annex G (informative) Example of measurement and reporting of completeness and thematic
accuracy .42
G.1 Introduction.42
G.2 Dataset description .42
G.3 Evaluation of data quality.47
G.4 Reporting quality results .50
Annex H (informative) Example of an aggregated data quality result.53
H.1 Introduction.53
H.2 Dataset description .53
H.3 Universe of discourse.54
H.4 Dataset.55
H.5 Aggregation of evaluation results and reporting.55
Annex I (normative) Reporting quality information in a quality evaluation report .57
I.1 Introduction.57
I.2 Quality evaluation report components.57
Annex J (informative) Aggregation of data quality results.61
J.1 Introduction.61
J.2 100 % pass/fail.61
J.3 Weighted pass/fail.61
J.4 Subset of results sufficient for product purpose.62
J.5 Maximum/minimum value.62
Bibliography.63
iv © ISO 2003 — All rights reserved
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies
(ISO member bodies). The work of preparing International Standards is normally carried out through ISO
technical committees. Each member body interested in a subject for which a technical committee has been
established has the right to be represented on that committee. International organizations, governmental and
non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the
International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of technical committees is to prepare International Standards. Draft International Standards
adopted by the technical committees are circulated to the member bodies for voting. Publication as an
International Standard requires approval by at least 75 % of the member bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO shall not be held responsible for identifying any or all such patent rights.
ISO 19114 was prepared by Technical Committee ISO/TC 211, Geographic information/Geomatics.
Introduction
For the purpose of evaluating the quality of a dataset, clearly defined procedures must be used in a consistent
manner. This enables data producers to express how well their product meets the criteria set forth in its
product specification and enables data users to establish the extent to which a dataset meets their
requirements. The quality of a dataset is described using two components: a quantitative component and a
non-quantitative component. The objective of this International Standard is to provide guidelines for evaluation
procedures of quantitative quality information for geographic data in accordance with the quality principles
described in ISO 19113. It also offers guidance on reporting quality information.
This International Standard recognizes that a data producer and a data user may view data quality from
different perspectives. Conformance quality levels can be set using the data producer’s product specification
or a data user’s data quality requirements. If the data user requires more data quality information than that
provided by the data producer, the data user may follow the data producer’s data quality evaluation process
flow to get the additional information. In this case, the data user requirements are treated as a product
specification for the purpose of using the data producer process flow.
The quality evaluation procedures described in this International Standard, when applied in accordance with
ISO 19113, provide a consistent and standard manner to determine and report the quality information in a
dataset.
vi © ISO 2003 — All rights reserved
INTERNATIONAL STANDARD ISO 19114:2003(E)
Geographic information — Quality evaluation procedures
1 Scope
This International Standard provides a framework of procedures for determining and evaluating quality that is
applicable to digital geographic datasets, consistent with the data quality principles defined in ISO 19113. It
also establishes a framework for evaluating and reporting data quality results, either as part of data quality
metadata only, or also as a quality evaluation report.
This International Standard is applicable to data producers when providing quality information on how well a
dataset conforms to the product specification, and to data users attempting to determine whether or not the
dataset contains data of sufficient quality to be fit for use in their particular applications.
Although this International Standard is applicable to all types of digital geographic data, its principles can be
extended to many other forms of geographic data such as maps, charts and textual documents.
2 Conformance
This International Standard defines three classes of conformance: one for quality evaluation procedures, one
for evaluating data quality, and one for reporting quality information. The abstract test suites for the three
classes of conformance are given in Annex A.
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
ISO 19113:2002, Geographic information — Quality principles
ISO 19115:2003, Geographic information — Metadata
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 19113 and ISO 19115 (some of
which are repeated for convenience) and the following apply.
4.1
conformance quality level
threshold value or set of threshold values for data quality results used to determine how well a dataset meets
the criteria set forth in its product specification or user requirements
4.2
dataset
identifiable collection of data
[ISO 19115]
NOTE A dataset may be a smaller grouping of data which, though limited by some constraint such as spatial extent
or feature type, is located physically within a larger dataset. For purposes of data quality evaluation, a dataset may be as
small as a single feature or feature attribute contained within a larger dataset.
4.3
dataset series
collection of datasets sharing the same product specification
[ISO 19115]
4.4
direct evaluation method
method of evaluating the quality of a dataset based on inspection of the items within the dataset
4.5
full inspection
inspection of every item in a dataset
NOTE Full inspection is also known as 100 % inspection.
4.6
indirect evaluation method
method of evaluating the quality of a dataset based on external knowledge
NOTE Examples of external knowledge are dataset lineage, such as production method or source data.
4.7
item
that which can be individually described or considered
[ISO 2859-1]
NOTE An item can be any part of a dataset, such as a feature, feature relationship, feature attribute, or combination
of these.
4.8
population
totality of items under consideration
[ISO 3534-2]
EXAMPLE 1 All points in a dataset.
EXAMPLE 2 Names of all roads in a certain geographic area.
4.9
reference data
data accepted as representing the universe of discourse, to be used as reference for direct external quality
evaluation methods
5 Abbreviated terms
ADQR aggregated data quality results
AQL acceptance quality limit [ISO 3534-2]
RMSE root mean square error
2 © ISO 2003 — All rights reserved
6 Process for evaluating data quality
6.1 General
A quality evaluation process may be used in different phases of a product life cycle, having different objectives
in each phase. The phases of the life cycle considered here are specification, production, delivery, use and
update. Annex B describes some specific dataset-related operations to which quality evaluation procedures
are applicable.
The process for evaluating data quality is a sequence of steps to produce and report a data quality result. A
quality evaluation process consists of the application of quality evaluation procedures to specific dataset-
related operations performed by the dataset producer and the dataset user.
Processes for evaluating data quality are applicable to static datasets and to dynamic datasets. Dynamic
datasets are datasets that receive updates so frequently that for all practical purposes they are continuously
changing. Annex C describes the application of the process to evaluate data quality to dynamic datasets.
6.2 Components of the process
6.2.1 Process flow
The quality evaluation process is a sequence of steps taken to produce a quality evaluation result. Figure 1
illustrates the process flow for evaluating and reporting data quality results.
Figure 1 — Evaluating and reporting data quality results
6.2.2 Process steps
Table 1 specifies the process steps.
Table 1 — Process steps
Process Action Description
step
1 Identify an applicable data quality The data quality element, data quality sub-element, and data quality
element, data quality sub-element, scope to be tested is identified in accordance with the requirements
and data quality scope of ISO 19113. This is repeated for as many different tests as required
by the product specification or user requirements.
2 Identify a data quality measure A data quality measure, data quality value type and, if applicable, a
data quality value unit is identified for each test to be performed.
Annex D provides examples of data quality measures for the data
quality elements and data quality sub-elements given in ISO 19113.
Annex D, by these examples, provides assistance to the user in
selection of a measure.
3 Select and apply a data quality A data quality evaluation method for each identified data quality
evaluation method measure is selected.
NOTE A spatial description of the results (achievable by spatial
interpolation of the results, graphical portrayal, etc.) might be useful,
corresponding not to a result, but to a different, although related, dataset.
4 Determine the data quality result A quantitative data quality result, a data quality value or set of data
quality values, a data quality value unit and a date are the output of
applying the method.
5 Determine conformance Whenever a conformance quality level has been specified in the
product specification or user requirements, the data quality result is
compared with it to determine conformance. A conformance data
quality result (pass-fail) is the comparison of the quantitative data
quality result with a conformance quality level.
7 Data quality evaluation methods
7.1 Classification of data quality evaluation methods
A data quality evaluation procedure is accomplished through the application of one or more data quality
evaluation methods. Data quality evaluation methods are divided into two main classes: direct and indirect.
Direct methods determine data quality through the comparison of the data with internal and/or external
reference information. Indirect methods infer or estimate data quality using information on the data, such as
lineage. The direct evaluation methods are further subclassified by the source of the information needed to
perform the evaluation. Figure 2 depicts this classification structure.
Figure 2 — Classification of data quality evaluation methods (informative)
4 © ISO 2003 — All rights reserved
7.2 Direct evaluation methods
7.2.1 Types of direct evaluation methods
The direct evaluation method is further subdivided into internal and external. All the data needed to perform an
internal direct data quality evaluation method are internal to the dataset being evaluated.
EXAMPLE 1 All the data necessary to perform a logical consistency test for topological consistency of boundary
closure resides in a topologically structured dataset.
External direct quality evaluation requires reference data external to the dataset being tested.
EXAMPLE 2 The data needed to perform a completeness test for the road names in a dataset requires another
information source of road names.
EXAMPLE 3 A positional accuracy test requires a reference dataset or a new survey.
7.2.2 Means of accomplishing direct evaluation
For both external and internal evaluation methods, there are two considerations, automated or non-automated
and full inspection or sampling.
Data quality elements and data quality sub-elements which are easily checked by automated means include
the following:
a) logical consistency: format consistency;
EXAMPLE Check data fields for positive entry.
b) logical consistency: topological consistency;
EXAMPLE Polygon closure.
c) logical consistency: domain consistency;
EXAMPLE Bounds violations, specified domain value violations.
d) completeness: omission;
EXAMPLE Comparison check of street names from another file.
e) completeness: commission;
EXAMPLE Comparison check of street names from another file.
f) temporal accuracy: temporal consistency.
EXAMPLE Check all records for appropriate range of dates.
7.2.3 Full inspection
Full inspection requires testing every item in the population specified by the data quality scope. Table 2
describes the procedure for full inspection that shall be used.
Table 2 — Procedure for full inspection
Procedure step Description
Define items An item is a minimum unit to be inspected. An item can be a feature, a
feature attribute or a feature relationship.
Inspect items in the data quality scope Inspect every item it the data quality scope.
NOTE Full inspection is most appropriate for small populations or for tests that can be accomplished by automated
means.
7.2.4 Sampling
Sampling requires testing sufficient items in the population in order to achieve a data quality result. Table 3
describes the sampling procedure that shall be used.
Table 3 — Sampling procedure
Procedure step Description
Define a sampling method Examples of sampling methods are given in Annex E. Those methods
include simple random sampling, stratified sampling (e.g. guided by
feature type, a feature relationship or an area), multistage sampling and
non-random sampling.
Define items An item is a minimum unit to be inspected. An item can be a feature, a
feature attribute or a feature relationship.
Divide data quality scope (population) into lots A lot is a collection of items in the data quality scope from which a
sample is drawn and inspected. Each lot should, as far as possible,
consist of items produced under the same conditions and at the same
time.
Divide lots to sampling units Sampling unit is the area of the lot where inspection is conducted.
Define the sampling ratio or sample size A sampling ratio gives information on how many items on average are
extracted for inspection from each lot.
Select sampling units Select required number of sampling units so that the sampling ratio or
sample size for items is fulfilled.
Inspect items in the sampling units Inspect every item in the sampling units.
The sampling procedure shall be reported in accordance with Clause 8.
The ISO 2859 series and ISO 3951-1 may be applied to sampling for evaluating conformance to a product
specification. These standards were originally developed for non-spatial use. Annex E of this International
Standard gives examples describing how to apply the ISO 2859 series and ISO 3951-1 and provides
guidelines on how to define samples and devise sampling methods, taking the geographic nature of the data
into account.
The reliability of the data quality result should be analysed when using sampling, especially when using small
sample sizes and methods other than simple random sampling.
7.3 Indirect evaluation method
The indirect evaluation method is a method of evaluating the quality of a dataset based on external knowledge.
This external knowledge may include, but is not limited to, data quality overview elements and other quality
reports on the dataset or data used to produce the dataset.
NOTE 1 This method is recommended only if direct evaluation methods cannot be used.
6 © ISO 2003 — All rights reserved
NOTE 2 Usage information records uses of a dataset. This is helpful when searching for datasets that have been
produced or used for specific purposes.
NOTE 3 Lineage information records information about the production and history of the dataset. It includes information
about, for example, source materials to produce a dataset or the production processes applied. This is useful when
determining the suitability of a dataset for a given use. An example is lineage metadata relating to a digital terrain model
file that has been created by means of stereo-correlation from images captured under certain conditions. Experience tells
the evaluator that the horizontal positional RMSE is 10 m for this type of imagery. Or, for example, lineage metadata of a
digitized 1:25 000 scale topographic map indicates conformance to a town planner’s requirements for a base map.
NOTE 4 Purpose information describes the purpose for which the dataset was produced. A purpose may be in support
of a specific requirement, or the dataset may be a general purpose dataset for several uses. This is useful when
identifying the possible value of a dataset.
7.4 Data quality evaluation examples
Examples of typical methods used and how they may be applied are described in Annexes F, G and H.
8 Reporting data quality evaluation information
8.1 Reporting as metadata
Quantitative quality results shall be reported as metadata in compliance with ISO 19115, which contains the
related model and data dictionary.
8.2 Reporting in a quality evaluation report
There are two conditions under which a quality evaluation report shall be produced:
a) when data quality results reported as metadata are only reported as pass/fail;
b) when aggregated data quality results are generated.
The report is required in the latter condition to explain how aggregation was done and how to interpret the
meaning of the aggregate result. However, a quality evaluation report may be created at any other time (such
as to provide more detail than reported as metadata) but a quality evaluation report cannot be used in lieu of
reporting as metadata.
A quality evaluation report shall be produced in compliance with Annex I which contains the relevant model
and data dictionary.
8.3 Reporting aggregated data quality result
When several quality results are aggregated into a single quality result for reporting the quality of a dataset,
the aggregated data quality result shall be reported as metadata and shall be included in the data quality
report. The data quality result shall be reported as type “aggregate”. Annex J describes the production of
aggregate data quality results and Annex H provides a production example.
Annex A
(normative)
Abstract test suites
A.1 Introduction
This annex defines three classes of conformance
quality evaluation procedure (A.2),
evaluating data quality (A.3), and
reporting data quality (A.4).
Any quality evaluation procedures claiming conformance with this International Standard shall pass all the
requirements given in A.2. Any evaluation of data quality claiming conformance with this International
Standard shall pass all the requirements given in A.3. Any report of data quality claiming conformance with
this International Standard shall pass all the requirements given in A.4.
NOTE All of the test cases are of test type “basic”.
A.2 Quality evaluation procedures
Abstract test suite for class 1 shall be as follows.
a) Test purpose: to assure the quality evaluation procedure was produced in accordance with this
International Standard.
b) Test method: pass all the requirements described in A.3 and A.4.
c) Reference: A.3 and A.4.
A.3 Evaluating data quality
Abstract test suite for class 2 shall be as follows.
a) Test purpose: to assure the quality evaluation procedure was produced in accordance with the quality
evaluation process in Clause 6.
b) Test method: compare the quality evaluation procedure with the quality evaluation as appropriate.
c) Reference: ISO 19114:2003, Clause 6.
A.4 Reporting data quality
Abstract test suite for class 3 shall be as follows.
a) Test purpose: to assure data quality has been reported in accordance with Clause 8.
b) Test method: compare the quality evaluation reported to assure data quality results were appropriately
reported in accordance with Clause 8 and applicable annexes.
c) Reference: ISO 19114:2003, Clause 8.
8 © ISO 2003 — All rights reserved
Annex B
(informative)
Uses of quality evaluation procedures
B.1 Introduction
Quality evaluation procedures may be used in different phases of a product life cycle. This annex provides
examples of stages of a product's life cycle during which quality evaluation procedures may be applied.
B.2 Development of a product specification or user requirements
When developing a product specification or user requirement, quality evaluation procedures may be used to
assist in establishing conformance quality levels that should be met by the final product. A product
specification or user requirement should include conformance quality levels for the dataset and quality
evaluation procedures to be applied during production and updating.
B.3 Quality control during dataset creation
At the production stage, the producer may apply quality evaluation procedures, either explicitly established or
not contained in the product specification, as part of the process of quality control. The description of the
applied quality evaluation procedures, when used for production quality control, should be reported as lineage
metadata including, but not necessarily limited to, the quality evaluation procedures applied, conformance
quality levels established and the results.
B.4 Inspection for conformance to a product specification
On completion of the production, a quality evaluation process is used to produce and report data quality
results. These results may be used to determine whether a dataset conforms to its product specification. If the
dataset passes inspection (composed of a set of quality evaluation procedures), the dataset is considered to
be ready for use. The results of the inspection operation should be reported in accordance with Clause 8.
The outcome of the inspection will be either acceptance or rejection of the dataset. If the dataset is rejected,
then after the data have been corrected, a new inspection will be required before the product can be deemed
to be in conformance with the product specification.
B.5 Evaluation of dataset conformance to user requirements
Quality evaluation procedures are used to establish the conformance quality levels for a dataset to meet a
user requirement. Indirect as well as direct methods may be used in analyses of dataset conformance to user
requirements. The results of the quality evaluation for conformance to user requirements may be reported as
usage metadata for the dataset.
B.6 Quality control during dataset update
Quality evaluation procedures are applied to dataset update operations, both to the items being used for
update and to benchmark the quality of the dataset after update has occurred. Guidance for the use of
ISO 19113 and this International Standard on dynamic datasets is given in Annex C.
Annex C
(informative)
Applying quality evaluation procedures to dynamic datasets
C.1 Introduction
This annex describes how quality evaluation procedures may be applied to dynamic datasets. Here dynamic
datasets are defined as datasets that receive updates so frequently that, for all practical purposes, they are
continuously being updated. For example, an online cadastre dataset may receive updates every few minutes.
There are basically two ways to determine and report the quality of a dynamic dataset: benchmark and
continuous.
C.2 Determining and reporting the quality of a dynamic dataset
C.2.1 Benchmark procedure
The benchmark procedure is based on the establishment of a suitable reporting frequency (e.g. weekly or tri-
monthly) and making a copy of the dataset at the reporting date. Then the copy is tested as if it were a static
dataset. This type of testing and reporting will provide the quality of the dataset as of the date/time of the copy.
C.2.2 Continuous procedure
The continuous procedure is based on testing the updates and evaluating the impact of the updates. This is
equivalent to embedding the quality evaluation procedures given in this International Standard into an
process-oriented procedure (e.g. that given in ISO 9001). Since this procedure only can provide the current
status of the quality of the updated items, it is necessary to combine both benchmark and continuous
procedures as described in C.3 in order to establish the quality of the updated database.
C.3 Establishing continuous quality evaluation procedures
C.3.1 Identify the parts
In accordance with the steps given in 6.2, identify applicable data quality elements and their associated data
quality sub-elements, data quality scopes, data quality measure, and conformance quality levels to be used in
the evaluation and reporting of the results.
C.3.2 Select the method to be applied
Select the data quality evaluation method to be applied. Then the evaluation will be on the updated feature
and the relationship of that feature with the others within the data quality scope. In continuous quality
evaluation procedures, only indirect or internal direct methods may be applied.
EXAMPLES
a) Is the update from a trusted source?
b) Does the update preserve topological consistency?
c) Does the address of the feature updated retain logical consistency?
10 © ISO 2003 — All rights reserved
C.3.3 Establish a dataset quality reference
Use the benchmark procedure to establish reference values for the quality of the dataset for the features and
feature attributes within scope to be checked during the continuous testing.
C.3.4 Integrate continuous tests into update process
Integrate the continuous tests into the update process flow so that each proposed update is tested and
accepted before it is introduced into the dataset.
C.3.5 Dynamically update data quality results
By integrating the continuous tests into the update process flow, each accepted update causes the current
quality results to be adjusted accordingly. This will allow for immediate reports on dataset quality to be
generated.
C.4 Periodically re-establish the reference quality of the dataset
All aspects of the quality of a dataset may not be tested through a continuous process-based operation. For
example, omission of features may not be found when only updated items are tested. The dataset should be
subject to periodic benchmark-type quality procedures.
Annex D
(informative)
Examples of data quality measures
D.1 Introduction
This annex provides simple examples of data quality measures for each data quality element and its
associated sub-elements, defined in ISO 19113, to demonstrate how the data quality components relate
during an evaluation operation. More detailed examples may be found in other annexes of this International
Standard.
For each data quality element and sub-element combination, an example data quality scope is given together
with example dataset parameters. Then three data quality measures are shown, each designed to
demonstrate a different way to evaluate quality. The examples are as complete as possible; a data quality
date and conformance quality level are given. Finally, an interpretation of the data quality result is given as an
example of the quality result meaning.
While the examples given in this annex are simple, it may be desirable to refer to them in profiles or other
documents. Therefore, this annex has a data quality measure identification code which relates the example to
the data quality element and data quality sub-element.
D.2 Relationship of the data quality components
Table D.1 gives the relationship of the data quality components.
In order to save space, each data quality component has been given a short name that will be used
throughout this annex.
12 © ISO 2003 — All rights reserved
Table D.1 — Relationship of data quality components
a
Data quality components Component domain Example
Short name
Data quality scope DQ_Scope Free text All items classified as houses
Data quality element DQ_Element Enumerated domain 1 – Completeness
1 – completeness data quality element
2 – logical consistency describing the presence or
3 – positional accuracy absence of features, their
4 – temporal accuracy attributes and their
5 – thematic accuracy relationships
Data quality subelement DQ_Subelement Enumerated domain 1 – Commission
(Dependent upon data quality excess data in the dataset
element)
EXAMPLE
Data quality measure DQ_Measure
Data quality measure DQ_MeasureDesc Free text Existence of excess items
description
Data quality measure DQ_MeasureID Enumerated domain 10101
identification code
Data quality evaluation method DQ_EvalMethod
Data quality evaluation DQ_EvalMethodType Enumerated domain 2 – external
method type 1 – internal (direct)
2 – external (direct)
3 – indirect
Data quality evaluation DQ_EvalMethodDesc Free text or citation (depends Compare count of items in
method description on data quality evaluation dataset against count of items
method type) in universe of discourse
Data quality result DQ_QualityResult
Data quality value type DQ_ValueType Enumerated domain 1 – Boolean variable
1 – Boolean variable
2 – number
3 – ratio
4 – percentage
5 – sample
6 – table
7 – binary image
8 – matrix
9 –
...










Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.
Loading comments...