ETSI TS 105 174-2-2 V1.1.1 (2009-10)

Technical Specification
Access, Terminals, Transmission and Multiplexing (ATTM);
Broadband Deployment - Energy Efficiency
and Key Performance Indicators;
Part 2: Network sites;
Sub-part 2: Data centres
2 ETSI TS 105 174-2-2 V1.1.1 (2009-10)

Reference
DTS/ATTM-02005-1-3
Keywords
access, cable, optical, site engineering
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2009.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 ETSI TS 105 174-2-2 V1.1.1 (2009-10)
Contents
Intellectual Property Rights . 6
Foreword . 6
Introduction . 6
1 Scope . 7
2 References . 8
2.1 Normative references . 8
2.2 Informative references . 9
3 Definitions and abbreviations . 9
3.1 Definitions . 9
3.2 Abbreviations . 11
4 Overview of data centres . 12
4.1 Types of data centres . 12
4.2 Tiering of data centres . 13
4.2.1 Tiers and criticality . 13
4.2.2 ANSI/TIA-942 . 13
4.2.3 Uptime Institute . 13
4.2.4 Criticality levels . 14
4.2.5 Tiers and costs . 14
4.3 Issues faced by data centres . 15
4.3.1 General . 15
4.3.2 Current issues. 16
4.3.2.1 Overview . 16
4.3.2.2 Principle issues . 16
4.3.2.3 Operator data centres. 17
4.3.3 Evolution and future trends . 17
4.3.3.1 Power and cooling demands of IT equipment . 17
4.3.3.2 New projects . 17
4.3.3.3 Data centre consolidation programmes . 18
4.3.3.4 Environmental impacts . 18
4.4 The new context . 18
4.4.1 Energy consumption and energy efficiency . 18
4.4.2 Factors impacting energy efficiency . 19
4.4.3 On-going initiatives . 19
5 Energy efficiency standards and metrics . 20
5.1 Review of activities outside ETSI . 20
5.1.1 EU Code of Conduct on Data Centres Energy Efficiency . 20
5.2 Energy consumption in data centres . 20
5.3 Energy Efficiency Key Performance Indicators (KPIs) . 21
5.3.1 Power Usage Effectiveness (PUE) or Data Centre Infrastructure Efficiency (DCIE) . 21
5.3.1.1 General . 21
5.3.1.2 PUE for new data centres . 22
5.3.1.3 PUE in existing data centres. 23
5.3.2 Other KPIs . 24
5.3.2.1 Energy efficiency KPI . 24
5.3.2.2 Consolidation KPI . 24
5.3.2.3 Data collection . 25
6 Increasing the energy efficiency of IT infrastructures. 25
6.1 General . 25
6.2 3 Tier Software Architecture Model . 25
6.3 Energy efficiency solutions . 26
6.3.1 Obsolete equipment . 26
6.3.2 Replacement equipment . 26
ETSI
4 ETSI TS 105 174-2-2 V1.1.1 (2009-10)
6.3.3 Power and capacity management . 26
6.3.3.1 General . 26
6.3.3.2 Power management . 26
6.3.3.2.1 Activation of basic power management features . 26
6.3.3.2.2 Activation of "sleep" mode . 27
6.3.3.2.3 Reduction of energy consumption of environmental control equipment . 28
6.3.3.3 Capacity management . 28
6.3.3.3.1 General . 28
6.3.3.3.2 Environmental capacity management . 29
6.3.3.3.3 Storage . 29
6.3.3.3.4 Servers . 29
6.3.3.3.5 On-demand scalability for on-line business. 29
6.3.4 Consolidation initiatives . 29
6.3.4.1 Consolidation of servers . 29
6.3.4.2 Physical consolidation . 30
6.3.4.2.1 The process . 30
6.3.4.2.2 The effects . 30
6.3.4.3 Virtualization . 30
6.3.4.3.1 The process . 30
6.3.4.3.2 The effects . 30
6.3.4.3.3 Reduction of energy consumption of IT infrastructure . 30
6.3.4.3.4 Reduction of energy consumption of environmental control equipment . 31
6.3.4.4 Logical consolidation . 31
6.3.4.4.1 The process . 31
6.3.4.4.2 The effects . 32
6.3.4.5 Application consolidation . 32
6.3.4.5.1 General . 32
6.3.4.5.2 The process . 32
6.3.4.5.3 The effects . 33
7 Reducing the energy demand of environmental control systems . 33
7.1 General . 33
7.2 Energy reduction solutions . 34
7.2.1 Measurement of thermal behaviour . 34
7.2.2 Improvement of cooling efficiency . 34
7.2.2.1 Zonal approaches to thermal isolation. 34
7.2.2.1.1 General . 34
7.2.2.1.2 Hot aisle and cold aisle segregation. 34
7.2.2.1.3 Cold aisle covers. 34
7.2.2.1.4 Segregation using curtains . 35
7.2.2.1.5 High density areas . 35
7.2.2.2 Reduction of thermal waste in cabinets/racks . 36
7.2.3 Modification of temperature and humidity . 37
7.2.3.1 General . 37
7.2.3.2 Results of experimentation . 37
7.2.3.3 Time before "system-shutdown" . 38
7.2.3.4 Restrictions on implementation . 39
7.2.4 Alternative cooling mechanisms . 39
7.2.4.1 Free cooling. 39
7.2.4.2 Direct liquid cooling . 41
7.2.4.3 Emerging technology (auto cooled chassis or chip-level cooling) . 41
7.2.4.3.1 Cooling-on-the-chip . 41
7.2.4.3.2 Auto-cooled chassis . 41
7.2.5 Enhancements of cooling systems . 41
8 Infrastructure requirements to optimize energy efficiency. 42
9 Improvement of energy efficiency of power distribution systems . 42
9.1 General . 42
9.2 Uninterruptible Power Supplies (UPS). 43
9.2.1 Efficiency . 43
9.2.2 Modular UPS . 44
9.3 Energy efficiency improvement solutions . 45
ETSI
5 ETSI TS 105 174-2-2 V1.1.1 (2009-10)
9.3.1 Measurement of energy efficiency of existing equipment . 45
9.3.2 Energy capacity management . 45
9.3.3 Review of policy . 45
9.3.3.1 General . 45
9.3.3.2 HVDC versus AC . 45
9.3.4 High efficiency distribution equipment . 45
9.3.5 Backup power . 46
10 Energy efficiency within existing data centres . 46
11 Energy efficiency within new data centres . 46
11.1 General . 46
11.2 Multi-disciplinary approach . 47
11.2.1 Location . 47
11.2.2 Energy sources . 47
11.2.2.1 Main energy . 47
11.2.2.2 Backup energy . 48
11.2.3 Building conception . 48
11.2.4 Internal design . 48
11.2.5 Energy and cooling . 49
11.2.6 IT infrastructure . 49
11.2.7 Software . 49
11.2.8 Processes . 50
12 Conformance . . 50
12.1 Existing data centres . 50
12.2 New data centres . 50
13 Recommendations . 51
13.1 Existing data centres . 51
13.1.1 General . 51
13.1.2 Reduction of PUE . 51
13.1.3 Reduction of energy consumption . 51
13.1.4 Optimum usage of existing resources . 51
13.1.5 High density areas . 51
13.2 New data centres . 52
13.2.1 General . 52
13.2.2 Location study . 52
13.2.3 Data centre construction . 52
13.2.3.1 External . 52
13.2.3.2 Internal . 52
13.2.4 Cooling . 53
13.2.4.1 systems . 53
13.2.4.2 Temperature control . 53
13.2.5 IT Infrastructure . 53
13.2.5.1 Architecture and policy . 53
13.2.5.2 Automation and capacity management . 54
13.2.6 Organization - processes . 54
14 Future opportunities . 55
14.1 General . 55
14.2 Energy . 55
14.3 Cooling . 55
14.4 Energy efficient IT hardware . 55
14.5 Software, tools and re-engineering . 56
14.6 Consolidation initiatives . 56
Annex A (informative): Indications of the effect of energy efficiency actions . 57
History . 58

ETSI
6 ETSI TS 105 174-2-2 V1.1.1 (2009-10)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Technical Specification (TS) has been produced by ETSI Technical Committee Access, Terminals, Transmission
and Multiplexing (ATTM).
The present document is part 2-2 of a multi-part deliverable. Full details of the entire series can be found in part 1 [13].
Introduction
The increasing interaction between the different elements of the Information Communication Technology (ICT) sector
(hardware, middleware, software and services) supports the concept of convergence in which:
• multi-service packages can be delivered over a common infrastructure;
• a variety of infrastructures is able to deliver these packages;
• a single multi-service-package may be delivered over different infrastructures.
As a result of this convergence, the development of new services, applications and content has resulted in an increased
demand for bandwidth, reliability, quality and performance, with a consequent increase in the demand for power which
has implications for cost and, in some cases, availability. It is therefore important to maximize the energy efficiency of
all the network elements necessary to deliver the required services.
New technologies and infrastructure strategies are expected to enable operators to decrease the energy consumption, for
a given level of service, of their existing and future infrastructures thus decreasing their costs. This requires a common
understanding among market participants that only standards can produce.
The present document is part 2-2 of a multi-part deliverable which has been produced by ETSI Technical Committee
Access, Terminals and Transmission, Multiplexing (ATTM) in close collaboration with CENELEC via the
Co-ordination Group on Installations and Cabling (CGIC). It offers a contribution to the required standardization
process by establishing an initial basis for work on ICT networks and transmission engineering, with active
collaboration from a number of other ETSI and CENELEC Technical Bodies. When complete, the multi-part
deliverable will contain information that has been jointly evolved to present developments in installations and
transmission implementation, and describing their progress towards energy efficiency in Next Generation Networks
(NGN).
In order to monitor the implementation and operation of energy efficient broadband deployment, the present document
also discusses Key Performance Indicators (KPI) for energy efficiency and focus on the possible consequences of
standardization of installations, cabling techniques and equipment. In particular, the study will investigate possibilities
and suggest solutions for development of processes for optimization in installation techniques and energy consumption.
ETSI
7 ETSI TS 105 174-2-2 V1.1.1 (2009-10)
1 Scope
The present document details measures which may be taken to improve the energy efficiency within operators sites and
data centres for broadband deployment. Clauses 2 and 3 contain references, definitions and abbreviations which relate
to this part; similar information will be included in the corresponding clauses of the other parts, thus ensuring that each
document can be used on a "stand-alone" basis.
Within the present document:
• clause 4 introduces data centre concepts including those specifically related to network operators;
• clause 5 develops the concept of Key Performance Indicators (KPI), introduced in TS 105 174-1 [13], to
enable consistent monitoring of energy efficiency;
• clause 6 details the approaches that may be employed to improve energy efficiency within the information
technology infrastructure;
• clause 7 details the approaches that may be employed to improve energy efficiency within the environmental
control systems;
• clause 8 details the approaches that may be employed to improve energy efficiency via the physical
infrastructure of the buildings;
• clause 9 details the approaches that may be employed to improve energy efficiency within the power
distribution system;
• clause 10 provides a summary of energy efficiency approaches within existing data centres;
• clause 11 provides a summary of energy efficiency approaches within new data centres and introduces wider
issues concerning their location;
• clause 12 contains the conformance mechanisms of the present document;
• clause 13 contains the recommendations of the present document;
• clause 14 introduces future opportunities for improvements of energy efficiency;
• annex A provides indications of the first order effect of applying the approaches outlined in clauses 6, 7 and 9.
This will enable the proper implementation of services, applications and content on an energy efficient infrastructure,
though it is not the goal of this multi-part deliverable to provide detailed standardized solutions for network
architecture.
The present document focuses on energy efficiency. The CO footprint is not taken in account in the present document.
Two separate aspects of energy efficiency are considered as shown in figure 1:
• actions to improve energy efficiency in existing data centres in the short or medium term;
• actions to improve energy efficiency in new data centres, in medium or long term.
The domains under study are:
• in the Information Technology (IT) infrastructure: all aspects of the technical infrastructure in the data centre,
including servers, storage arrays, backup libraries and network equipment including routers, switches, etc.;
• in the IT operational strategy: all consolidation initiatives, such as virtualization, physical or logical
consolidations, usage of specific software and processes;
• in the technical environment: all aspects concerning energy usage, cooling and, more generally, all disciplines
involved in the technical environment of the data centre.
ETSI
8 ETSI TS 105 174-2-2 V1.1.1 (2009-10)

Figure 1: Aspects of data centres under consideration
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific.
• For a specific reference, subsequent revisions do not apply.
• Non-specific reference may be made only to a complete document or a part thereof and only in the following
cases:
- if it is accepted that it will be possible to use all future changes of the referenced document for the
purposes of the referring document;
- for informative references.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are indispensable for the application of the present document. For dated
references, only the edition cited applies. For non-specific references, the latest edition of the referenced document
(including any amendments) applies.
[1] ANSI/TIA-942: "Telecommunications Infrastructure Standard for Data Centres".
[2] Uptime Institute: "Tier Classifications Define Site Infrastructure Performance".
[3] Johannesburg: "Datacenter Dynamics Research Key Findings" August 2008.
[4] European Commission: "DG-JRC Code of Conduct on Data Centres Energy Efficiency".
[5] "Best Practices for the EU Code of Conduct on Data Centres".
ETSI
9 ETSI TS 105 174-2-2 V1.1.1 (2009-10)
[6] CENELEC EN 50173-2: "Information technology - Generic cabling systems - Part 2: Office
premises".
[7] CENELEC EN 50173-5: "Information technology - Generic cabling systems - Part 5: Data
centres".
[8] CENELEC EN 50174-1: "Information technology - Cabling installation - Part 1: Installation
specification and quality assurance".
[9] CENELEC EN 50174-2: "Information technology - Cabling installation - Part 2: Installation
planning and practices inside buildings".
[10] High performance buildings: "UPS report (Ecos Consulting-Epri Solutions)".
[11] ETSI EN 300 019-1-3: "Environmental Engineering (EE); Environmental conditions and
environmental tests for telecommunications equipment; Part 1-3: Classification of environmental
conditions; Stationary use at weatherprotected locations".
[12] ETSI EN 300 132-3: "Environmental Engineering (EE); Power supply interface at the input to
telecommunications equipment; Part 3: Operated by rectified current source, alternating current
source or direct current source up to 400 V".
[13] ETSI TS 105 174-1: "Access, Terminals, Transmission and Multiplexing (ATTM); Broadband
Deployment - Energy Efficiency and Key Performance Indicators; Part 1: Overview, common and
generic aspects".
2.2 Informative references
The following referenced documents are not essential to the use of the present document but they assist the user with
regard to a particular subject area. For non-specific references, the latest version of the referenced document (including
any amendments) applies.
[i.1] ETSI TR 102 489: "Environmental Engineering (EE); European telecommunications standard for
equipment practice; Thermal Management Guidance for equipment and its deployment".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
application: single program or a set of several programs executing a function or a service
availability: time or period during the application or the service has to be operational
NOTE: Availability is one of the criticality criteria.
blade server: server chassis housing multiple thin, modular electronic circuit boards, known as server blades
NOTE: Each blade is a server in its own right, often dedicated to a single application. The blades are literally
servers on a card, containing processors, memory, integrated network controllers, an optional fibre
channel host bus adaptor (HBA) and other input/output (IO) ports.
computer room: closed, secured and environmentally controlled room in which IT equipment is operating
criticality: level given to an application or service, linked to the impact for the enterprise in case of crash
NOTE: More the impact is strong, more the application or service is critical.
data centre: centralized repository for the storage, management, and dissemination of data and information organized
around a particular body of knowledge or pertaining to a particular business
ETSI
10 ETSI TS 105 174-2-2 V1.1.1 (2009-10)
Data Centre Infrastructure Efficiency (DCIE): reciprocal of the PUE, that is "IT equipment power" divided by "total
facility power", expressed as a percentage
NOTE: DCIE improves as it approaches 100 %.
Disaster Recovery Plan (DRP): all process (technical, organization, people) to launch in case of continuity disruption
disk array: cabinet containing physical disks
energy efficiency: search in existing DC, or for new future DC, of all tracks and actions allowing minimizing energy
needs and costs
NOTE: Key drivers are Economic to decrease the energy bill by increasing the efficiency of all equipment and
minimize power loss.
green data centre: in addition to energy efficiency, the "Green" approach will focus on carbon footprint
NOTE 1: Energy Efficiency is one way, to decrease CO emissions, but it is not the only one.
NOTE 2: More "sustainable development" objective than economic, the key indicator is carbon footprint. Today,
this concept is not still clearly defined, especially if we now that data centres are not directly producers of
CO , but indirectly, due to their energy needs. If the source of power is becoming from renewable
energies (hydraulic, solar, etc.) or nuclear (not so green for earth, but not producing CO ) the carbon
footprint of the datacenter is low. But if energy is becoming from coal, or fuel the CO emissions are
high.
information technology equipment: equipment such as computers, servers, mainframes, calculators and all storage
devices as arrays, libraries, tape robots together with routers and switches within the local area networks
IT equipment power: total power needed for operate servers, racks, disk arrays, libraries, network telecommunications
equipment (such as routers and switches), equipment used for monitoring the data centre (PC, laptops, terminals and
workstations) and network telecommunications-specific equipment (such as DSLAM and BTS)
logical consolidation ratio: number of application instances per operating system image
logical server: one single instance of operating system
mainframe: high-performance computer used for large-scale computing purposes that require greater availability and
security than a smaller-scale machine can offer
network telecommunications equipment: equipment providing direct connection to core and/or access networks
including switches, DSLAM, BTS
operator site: premises accommodating network telecommunications equipment providing direct connection to the
core and access networks and which may also accommodate information technology equipment
physical server: box containing supplies for energy, mother board, central processing unit, memory, slots
Power Usage Effectiveness (PUE): metric used to determine the energy efficiency of a data centre that is determined
by "Total facility power" divided by "IT equipment power", expressed as a ratio (PUE is expressed as a ratio, with
overall efficiency improving as the quotient decreases toward 1)
Recovery Point Objective (RPO): maximum allowed data loss
Recovery Time Objective (RTO): maximum authorized time during application or service can be stopped
server: computer program that provides services to other computer programs (and their users) in the same or other
computers
total computing load: total computing power in the data centre, that can be evaluated by taking vendors specifications
of computational power of each model of server multiplied by the number of servers (transactions per minute is one
measure of total computing power)
total facility power: total power used by all power delivery components (such as uninterruptible power supplies,
switches, power distribution units, batteries and transformers), cooling system components (such as chillers, computer
room air conditioning units, pumps, fans, engines) and the non-technical energy (such as building lighting)
ETSI
11 ETSI TS 105 174-2-2 V1.1.1 (2009-10)
TPC Benchmark C (TPC-C): On-Line Transaction Processing (OLTP) benchmark measured in transactions per
minute (TPMc)
NOTE 1: TPC-C is more complex than previous OLTP benchmarks such as TPC-A because of its multiple
transaction types, more complex database and overall execution structure. TPC-C involves a mix of five
concurrent transactions of different types and complexity either executed on-line or queued for deferred
execution. The database comprises nine types of tables with a wide r
...