ISO/IEC 23544:2021

INTERNATIONAL ISO/IEC
STANDARD 23544
First edition
2021-06
Information Technology — Data
centres — Application Platform
Energy Effectiveness (APEE)
Technologies de l'information — Centres de traitement de données —
Efficacité énergétique des plateformes d'applications (APEE)
Reference number
ISO/IEC 23544:2021(E)
©
ISO/IEC 2021

---------------------- Page: 1 ----------------------
ISO/IEC 23544:2021(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 23544:2021(E)

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms, definitions and abbreviated terms . 1
3.1 Terms and definitions . 1
3.2 Abbreviated terms . 3
4 Relevance of APEE . 3
5 Determination of APEE . 4
6 Determination of application outcome and energy consumption of the application
platform for APEE . 4
6.1 Prerequisites for the benchmark and application platform . 4
6.2 Determination of application outcome . 5
6.3 Determination of energy consumption of the application platform. 5
6.3.1 Requirements for obtaining energy consumption of the application platform. 5
6.3.2 Requirements for the measurement method for obtaining energy consumption . 6
7 Reporting of APEE . 6
7.1 Requirements . 6
7.2 Recommendations . 7
Annex A (informative) Use cases of APEE . 8
Annex B (informative) Example of APEE calculation .10
Annex C (informative) Example of an APEE report .11
Annex D (informative) Examples of acceptable benchmarks .21
Bibliography .23
© ISO/IEC 2021 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC 23544:2021(E)

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members
_experts/ refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/
iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 39, Sustainability, IT & Data Centres.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national
-committees.
iv © ISO/IEC 2021 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC 23544:2021(E)

Introduction
The growth of the Internet of Things (IoT) has resulted in the storage of increasingly large amounts
of data in data centres and the increased utilization of this data by technologies such as artificial
intelligence (AI). As a result, energy consumption of data centres is also increasing. In this recent
utilization of large amounts of data by technologies such as AI, various kinds of data are combined and
analysed, and processing requests for such combinations and analysis are also increasing. Middleware,
such as a database management system (DBMS), is also becoming more important for this data
utilization. The energy effectiveness of application platforms can be greatly improved through the
selection of middleware.
In addition to improving the energy effectiveness of target IT equipment itself, it is necessary to improve
application platform energy effectiveness by choosing an optimally energy-effective combination of
target IT equipment, operating systems and middleware.
Although there are several KPIs for the energy effectiveness of target IT equipment itself (for example,
ISO/IEC 30134-4, ISO/IEC 21836), there have previously been no KPIs for energy effectiveness to
calculate the energy effectiveness of a combination of target IT equipment, operating systems and
middleware. This document, therefore, introduces a KPI for the energy effectiveness of an entire
application platform. A typical use case of application platform energy effectiveness (APEE) is a
criterion for procuring an energy efficient application platform for an IT service.
This KPI provides assistance in selecting an optimal application platform for energy effectiveness.
This KPI does not apply to the energy effectiveness of an entire data centre. The colloquial term of
"efficiency" is commonly used in regional programmes and "effectiveness" can be referenced as
"efficiency" in those programmes.
© ISO/IEC 2021 – All rights reserved v

---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO/IEC 23544:2021(E)
Information Technology — Data centres — Application
Platform Energy Effectiveness (APEE)
1 Scope
This document specifies application platform energy effectiveness (APEE) as a Key Performance
Indicator (KPI) which quantifies the energy effectiveness of an application platform for an IT service in
data centres. This KPI evaluates the energy consumption of an application platform prior to deployment.
The purpose of this KPI is to measure the energy effectiveness of a set of target IT equipment, operating
systems and middleware, to enable the selection of an energy effective IT stack.
This document specifies a formula for calculating APEE and definitions of components of the formula.
This document specifies a measurement method for assessing and reporting the energy effectiveness of
an application platform.
This document also specifies requirements for benchmarks to be used for APEE and requirements for
reporting.
The following topics are outside of the scope of this document:
1) KPIs intended to solely evaluate the energy effectiveness of target IT equipment hardware,
2) energy effectiveness of data centre facilities.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 21836:2020, Information technology — Data centres — Server energy effectiveness metric
ISO/IEC 21878, Information technology — Security techniques — Security guidelines for design and
implementation of virtualized servers
3 Terms, definitions and abbreviated terms
3.1 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 21878 and the following
apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
© ISO/IEC 2021 – All rights reserved 1

---------------------- Page: 6 ----------------------
ISO/IEC 23544:2021(E)

3.1.1
application platform
platform composed of target IT equipment (3.1.4), operating systems, and middleware (3.1.3) with the
purpose of providing an IT service (3.1.5)
Note 1 to entry: An application platform is an environment for executing software applications (3.1.2) such as
ERP and CRM software. Therefore, a software application itself is not included as part of an application platform.
3.1.2
software application
computer program for implementing an IT service (3.1.5)
Note 1 to entry: A software application is usually developed independently for each IT service.
3.1.3
middleware
software layer between an operating system and the software applications (3.1.2)
Note 1 to entry: Examples of middleware include DBMS, web server software, application server software,
transaction monitor.
Note 2 to entry: An operating system is the software directly managing the hardware, e.g. conventional operating
system, hypervisor, etc.
[SOURCE: ISO/IEC/IEEE 24765:2017, 3.2459, modified – Notes 1 and 2 to entry have been added.]
3.1.4
target IT equipment
server, storage and network equipment that are components of an application platform (3.1.1)
3.1.5
IT service
service that provides an outcome to its users via information technology
3.1.6
application outcome
information provided by an IT service (3.1.5) to its users
Note 1 to entry: An example of an application outcome is a set of reports to support decision-making at retail
outlets (such as sales analysis reports, and stock status reports).
3.1.7
measurement period
duration in which the benchmark performs the intended data processing
Note 1 to entry: Typically, the measurement period(s) is/are the only portion of the benchmark execution which
is/are used to calculate the benchmark score.
Note 2 to entry: The intended data processing routine is a routine including the benchmark calculation.
1)
Note 3 to entry: In the case of using TPC-H , the measurement target of TPC-H is a performance for executing
SQL queries, so a period of executing SQL queries is a measurement period. Other periods, such as a data loading,
are not a measurement period.
3.1.8
benchmark identifier
string which identifies a specific benchmark, which consists of the name and the major version (3.1.9),
when applicable
1) TPC-H, TPC-E, and TPCx-IoT are trademarks of the Transaction Processing Performance Council. This
information is given for the convenience of users of this document and does not constitute an endorsement by ISO/
IEC of the product named. Equivalent products may be used if they can be shown to lead to the same results.
2 © ISO/IEC 2021 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC 23544:2021(E)

3.1.9
version
particular form or variation of a resource that differs from other instantiations of the resource in at
least one aspect or item of information
Note 1 to entry: The first digit of a version is called the major version, and any numbers right of that digit is called
a minor version.
Note 2 to entry: Example: in the version designated as “v1.2.3,” the Major Version identification is “1”, and the
Minor Version identification is “2.3”.
[SOURCE: ISO 24619:2011, 3.1.9, modified — Notes 1 and 2 to entry added.]
3.1.10
internal power supply
independent device that converts commercial AC power to DC power as required inside target IT
equipment (3.1.4)
3.2 Abbreviated terms
AC alternating current
AI artificial intelligence
APEE application platform energy effectiveness
CRM customer relationship management
DBMS database management system
DC direct current
EAP energy consumption of application platform during the measurement periods
ERP enterprise resource planning
FC fibre channel
IoT Internet of Things
NoAO number of application outcomes during an APEE measurement
OS operating system
RDBMS relational database management system
4 Relevance of APEE
Many kinds of IT services are provided by using data centres, and it is necessary to conserve the
amount of energy used by these IT services. A lot of IT services use various technologies, such as AI, IoT
and big data technologies. For each of these technologies, there are several combinations of target IT
equipment, OSs and middleware that can be used to provide IT services. These combinations of target
IT equipment, OSs and middleware are called an application platform.
Even if the results provided by an IT service are the same, if the application platform used to provide
the IT service is different, the energy consumption is also different. For example, when the same query
is executed on different DBMSs, because the processing method and processing time are different for
each DBMS, energy consumption also differs.
© ISO/IEC 2021 – All rights reserved 3

---------------------- Page: 8 ----------------------
ISO/IEC 23544:2021(E)

In order to choose the optimum application platform for energy effectiveness, a KPI for measuring the
energy effectiveness of an application platform is necessary. There are KPIs for measuring the energy
effectiveness of target IT equipment only, and ISO/IEC 21836 is one of KPIs for a server. APEE is a KPI
for measuring the energy effectiveness of an application platform. Use cases of APEE are described in
Annex A.
NOTE APEE is a KPI for measuring the energy effectiveness of an application platform for a particular IT
service, not the energy effectiveness of an entire data centre. Therefore, APEE is not suitable for use in regulations
that apply to an entire data centre.
5 Determination of APEE
APEE shall be determined and described by Formula (1):
c = a / b (1)
where
a is the number of application outcomes during an APEE measurement (NoAO) (see Clause 6);
b is the energy consumption (kWh) of the application platform during the measurement
periods (EAP) (see Clause 6);
c is the APEE.
The unit dimension of APEE is (application outcome(s)/kWh). The APEE value should have at least 2
significant digits.
Reporting of data for APEE is described in Clause 7.
An example of APEE calculation is given in Annex B.
6 Determination of application outcome and energy consumption of the
application platform for APEE
6.1 Prerequisites for the benchmark and application platform
The following prerequisites shall apply when measuring the APEE of an application platform that
provides an IT service:
a) The measurer determines one benchmark to be used for the calculation of APEE. Benchmarks shall
meet the following requirements. Examples of benchmarks that can be used for APEE measurement
are given in Annex D.
1) The benchmark is representative of the target IT service.
2) The start and the end of the execution of the benchmark are clear.
3) All iterations of the benchmark run produce comparable results.
4) An application outcome defined by the benchmark is identical regardless of how many times it
is executed on any application platform.
NOTE 1 In the case of TPC-H as an example of an application outcome, it defines a set of results of
business-oriented ad-hoc queries as an application outcome of the execution. A performance metric
such as throughput and processing time is not an application outcome.
4 © ISO/IEC 2021 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC 23544:2021(E)

EXAMPLE 1 A benchmark that executes transactions while increasing the multiplicity until
the system cannot keep the specified response time in order to obtain maximum multiplicity is not
applicable because the processes of measurement target for these benchmarks have the potential to
change from run to run.
5) Start date and time, and end date and time of all the measurement period(s) are clear.
NOTE 2 Some benchmarks have multiple measurement periods in one benchmark execution.
EXAMPLE 2 (Example of clarification) A benchmark has warmup and cooldown phases which do not
effort a benchmark performance score. It records start date and time and end date and time of these
phases in a log file, so these phases are distinguishable from measurement period(s).
6) Measurement period start and stop time are measured in seconds or higher precision.
7) To ensure that the result is verifiable, the benchmark is made publicly available.
b) The measurer determines the target IT equipment, OSs and middleware that are to serve as
the elements of the application platform. The application platform shall meet the following
requirements:
1) The application platform contains all of the target IT equipment necessary to execute the
benchmark.
2) Every target IT equipment component can be measured for energy consumption.
6.2 Determination of application outcome
For the determination of application outcome, the following terms are introduced:
— One application outcome is defined as one valid execution of the benchmark.
The following requirements for measurement shall be met:
a) The benchmark is executed at least one time. If the benchmark is executed multiple times, the
number of benchmark executions is a positive integer.
NOTE The tester can interact with the server between benchmark iterations, typically to calculate the
total measured time for benchmarks which include unmeasured periods.
b) If the total time of the whole measurement period is less than one hour, the benchmark is repeatedly
executed until the total time of the whole measurement period exceeds one hour.
c) If the benchmark has a setting for official or valid or reportable results for publication or use in
comparisons, this mode shall be used for APEE comparisons.
6.3 Determination of energy consumption of the application platform
6.3.1 Requirements for obtaining energy consumption of the application platform
Energy consumption of the application platform shall be obtained by the sum of energy consumption
(kWh) of all the target IT equipment in the application platform during all benchmark measurement
period(s) of the APEE measurement. Non-benchmark software should be minimal during the
measurement period(s). Execution of non-benchmark software on the application platform shall not
invalidate energy results. Energy measurements during benchmark execution shall not be adjusted to
remove the effects of non-benchmark software.
NOTE The application platform is usually used solely by the benchmark during the measuring period for
accuracy.
The energy consumption of each item of target IT equipment in the application platform is obtained by
the measurement method. Details of this method are given in 6.3.2.
© ISO/IEC 2021 – All rights reserved 5

---------------------- Page: 10 ----------------------
ISO/IEC 23544:2021(E)

6.3.2 Requirements for the measurement method for obtaining energy consumption
The measurement method shall meet the following requirements. The measurer shall replace “SEEM”
or “SERTv2” with “APEE” and “SUT” with “application platform” in the text of the SEEM referenced
below and apply.
a) Environment
1) ISO/IEC 21836:2020, 7.2 applies for measurement environment.
2) ISO/IEC 21836:2020, 7.4 applies for power requirements.
b) Power measurement
1) One or more power analysers shall be used to measure the entire energy consumption of the
application platform.
2) Maximum sampling interval of the power analysers shall be at least one sample per second.
3) ISO/IEC 21836:2020, 7.3 and 9.2.2.2 apply for power analyser requirements.
4) Power analysers shall comply with ISO/IEC 21836:2020, 9.2.2.3 and 9.2.2.4.
5) The following requirements shall be preferentially followed over the above ISO/IEC 21836
requirements.
i) The energy consumption shall be measured by using one or more power analysers.
ii) All clocks of all power analysers and target IT equipment shall be synchronized such that
time stamps are aligned within 1 second.
NOTE This requirement is intended to support measurements of the energy consumption of
multiple target IT equipment. ISO/IEC 21836 targets a server, but this document targets multiple target
IT equipment.
c) Temperature measurement
1) The instantaneous temperature of an inlet of one typical element of the application platform
shall be measured by a temperature sensor both within 1 minute of the start time of the
benchmark execution and within 1 minute of the end time of the benchmark execution.
2) ISO/IEC 21836:2020, 9.2.3 applies for temperature sensor requirements.
d) Approved measuring instrument
1) ISO/IEC 21836:2020, Annex B, Table 4 and Table 5 apply for pre-approved power analysers and
temperature sensors.
7 Reporting of APEE
7.1 Requirements
When presenting an APEE, both the benchmark identifier and the APEE value shall be presented, and
the benchmark identifier shall be written in parentheses as follows:
APEE(benchmark identifier)
For all benchmarks which include version numbers, the corresponding benchmark identifier shall
include the benchmark's major version number.
EXAMPLE APEE is written as APEE(TPC-H v2) when TPC-H version 2.17.3 is used as the benchmark.
6 © ISO/IEC 2021 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/IEC 23544:2021(E)

When presenting APEE, APEE shall be presented along with a report or link to a report. The report shall
include at least the following information:
a) APEE value
b) NoAO value and EAP value
c) start date and time of measurement, and end date and time of measurement
d) start date and time of each measurement period, and end date and time of each measurement
period
e) target IT service
f) details of the selected application platform:
1) vendor names and model names of target IT equipment;
2) vendor names, model names, and version numbers of operating systems and middleware;
3) system configuration diagram of the target IT equipment, operating systems, and middleware;
NOTE 1 For an example of a configuration diagram, see Annex C.
4) all target IT equipment configuration and software settings that affect NoAO and EAP values
or are required to reproduce the results shall be documented, including load range, power
management settings, and virtualization resource allocation configuration.
NOTE 2 Examples of load range in DBMS are database size and transaction throughput.
g) information about the selected benchmark:
1) name and version;
2) reference to the specifications of the benchmark;
3) benchmark settings descriptions and values used for all benchmark settings that affect NoAO
and EAP values.
h) power supply information (e.g. power supply mode, power supplies 80 plus rating, AC/DC, voltage,
frequency, and phase)
i) instantaneous temperatures of the inlet at start and end of benchmark execution, and location of
the temperature sensor
j) information about the measurement method:
1) vendor names and model names of the selected power analysers;
2) physical locations of the power analyser(s) between the power source and the measured
systems, including servers, storage, and network equipment;
3) any deviations from the measurement procedure and any unusual features observed.
k) number, publication year, and title of this document, i.e. ISO/IEC 23544:2021, Information
Technology — Data centres — Application Platform Energy Effectiveness (APEE)
7.2 Recommendations
The report should include the following information:
a) purpose of measuring of the APEE value;
b) reason for selecting the benchmark.
© ISO/IEC 2021 – All rights reserved 7

---------------------- Page: 12 ----------------------
ISO/IEC 23544:2021(E)

Annex A
(informative)

Use cases of APEE
A.1 Introduction
The life cycle of an IT service consists of the following phases:
a) planning,
b) procurement,
c) operation,
d) update,
e) disposal.
APEE can be effectively used in the procurement phase. This annex provides two use cases for the
procurement phase, from the perspective of the party procuring the IT service and from the perspective
of the party proposing the IT service. These parties fulfil the following roles in the use cases described
in this annex. The following description of characters is used only in this Annex A.
— IT service provider: entity that provides IT services to end users.
— System integrator: entity that builds an IT system that implements IT services by combining target
IT equipment and software.
A.2 Use cases
This annex provides the following two use cases.
Use case 1: Procurement of an IT service — A case where the IT service provider uses APEE as a
criterion for energy effectiveness when narrowing down candidates from among multiple system
integrator proposals.
a) The service provider chooses one benchmark, which is representative of the IT service in question,
and informs multiple system integrators of this benchmark, as shown in Annex D.
b) Each system integrator specifies the middleware and builds an application platform.
c) On their respective application platforms, each system integrator executes the benchmark selected
by the service provider.
d) The system integrator measures the energy consumption of the application platform, and calculates
the APEE value.
e) Each system integrator informs the service provider of the APEE value.
f) The service provider receives the APEE value from the multiple system integrators, and uses the
APEE value, as an expression of energy effectiveness, to narrow down candidates from among
multiple system integrator proposals.
8 © ISO/IEC 2021 – All rights reserved

---------------------- Page: 13 ----------------------
IS
...