ETSI TR 103 527 V1.1.1 (2018-07)

ETSI TR 103 527 V1.1.1 (2018-07)






TECHNICAL REPORT
SmartM2M;
Virtualized IoT Architectures with Cloud Back-ends

---------------------- Page: 1 ----------------------
2 ETSI TR 103 527 V1.1.1 (2018-07)



Reference
DTR/SmartM2M-103527
Keywords
cloud, IoT, virtualisation

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2018.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TR 103 527 V1.1.1 (2018-07)
Contents
Intellectual Property Rights . 6
Foreword . 6
Modal verbs terminology . 6
Introduction . 6
1 Scope . 8
2 References . 8
2.1 Normative references . 8
2.2 Informative references . 8
3 Definitions and abbreviations . 9
3.1 Definitions . 9
3.2 Abbreviations . 9
4 Rationale for IoT Virtualization . 10
4.1 IoT: towards massive deployments . 10
4.2 Cloud Computing and Virtualization . 10
4.3 The new challenge: combining IoT and Cloud Computing . 11
4.4 Content of the report. 11
5 Some use cases for IoT Virtualization. 12
5.1 Introduction . 12
5.2 Horizontal up and down Auto-Scaling . 12
5.3 No single point of failure . 13
5.4 Data privacy . 13
5.5 The use case selected as a proof-of-concept . 14
6 Cloud Computing features for IoT Virtualization . 15
6.1 Introduction . 15
6.2 Functional requirements . 15
6.2.1 Introduction. 15
6.2.2 Multi-tenancy . 15
6.2.2.1 Definition . 15
6.2.2.2 Comparison with multi-instance architectures . 15
6.2.3 Massive Data processing . 16
6.3 Non-functional requirements . 16
6.3.1 High-throughput . 16
6.3.2 High-availability . 17
6.3.3 Low latency . 18
6.3.3.1 Requirements . 18
6.3.3.2 MapReduce . 18
6.3.3.3 In Memory Databases . 19
6.3.3.4 Edge Computing . 19
6.3.4 Security . 20
6.4 Features in support of virtualized IoT implementations . 20
6.4.1 Microservices . 20
6.4.1.1 Definition . 20
6.4.1.2 Comparison to monolithic architectures. 21
6.4.1.3 Impact on IoT solutions . 21
6.4.1.4 Scaling microservices. 21
6.4.1.5 Providing persistency for microservices . 22
6.4.1.6 Security for microservices . 23
6.4.2 Inter-Process Communication (IPC) in microservices architecture . 23
6.4.2.1 Communication Mechanisms . 23
6.4.2.2 Synchronous IPC communications: RESTful communication . 23
6.4.2.3 Asynchronous IPC communications: Messaging . 24
6.4.2.4 Hybrid IPC communications . 24
ETSI

---------------------- Page: 3 ----------------------
4 ETSI TR 103 527 V1.1.1 (2018-07)
7 Implications of IoT virtualization . 25
7.1 Introduction . 25
7.2 Microservices for IoT Virtualization . 25
7.2.1 Microservices Architecture . 25
7.2.2 The Microservices Architecture in practice: an example . 26
7.2.3 Relationship of the microservice service HLA to oneM2M . 27
7.3 One High-Level Architecture for IoT Virtualization . 30
7.3.1 Functional Architecture for IoT Virtualization . 30
7.3.2 HLA for IoT Virtualization and oneM2M HLA . 30
8 Conclusions . 33
8.1 Implications . 33
8.2 Lessons Learned . 34
8.3 Recommendations to oneM2M . 34
Annex A: Relationship to big data . 35
Annex B: Relationship with NFV . 38
B.0 Introduction . 38
B.1 Virtualization in the NFV Architecture . 38
B.2 The NFV architecture and the Microservice-based HLA . 39
Annex C: Change History . 41
History . 42


ETSI

---------------------- Page: 4 ----------------------
5 ETSI TR 103 527 V1.1.1 (2018-07)
List of figures
Figure 1: Options for adoption of Cloud Native solutions .11
Figure 2: Batch and Streaming data processing .16
Figure 3: Achieving high throughput processing of data sets .17
Figure 4: The MapReduce Concept .18
Figure 5: Device Edge .19
Figure 6: Cloud Edge .20
Figure 7: RESTful IPC .23
Figure 8: Asynchronous Messaging IPC .24
Figure 9: Hybrid IPC communications .24
Figure 10: Microservices Architecture for IoT Virtualization .25
Figure 11: Message Flow Example .27
Figure 12: Common Services Functions defined by oneM2M .28
Figure 13: Comparison between the microservices architecture and oneM2M CSF .29
Figure 14: A High-Level Architecture for IoT Virtualization .30
Figure 15: Mapping the Microservice Architecture and oneM2M Common Service Entities .31
Figure 16: An example of implementation options of the microservices HLA .32
Figure A.1: Passive IoT fault detection and isolation module .36
Figure A.2: Fault detection: Outlier data-point .36
Figure A.3: Fault detection: Spike behaviour .37
Figure B.1: High Level NFV Framework .39


ETSI

---------------------- Page: 5 ----------------------
6 ETSI TR 103 527 V1.1.1 (2018-07)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Smart Machine-to-Machine
communications (SmartM2M).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
In addition to interoperability and security that are two recognized key enablers to the development of large IoT
systems, a new one is emerging as another key condition of success: virtualization. The deployment of IoT systems will
occur not just within closed and secure administrative domains but also over architectures that support the dynamic
usage of resources that are provided by virtualization techniques over cloud back-ends.
This new challenge for IoT requires that the elements of an IoT system can work in a fully interoperable, secure and
dynamically configurable manner with other elements (devices, gateways, storage, etc.) that are deployed in different
operational and contractual conditions. To this extent, the current architectures of IoT will have to be aligned with those
that support the deployment of cloud-based systems (private, public, etc.).
Moreover, these architectures will have to support very diverse and often stringent non-functional requirements such as
scalability, reliability, fault tolerance, massive data, security. This will require very flexible architectures for the
elements (e.g. the application servers) that will support the virtualized IoT services, as well as very efficient and highly
modular implementations that will make a massive usage of Open Source components.
These architectures and these implementations form a new approach to IoT systems and the solutions that the present
document investigates also should be validated: to this extent, a Proof-of-Concept implementation involving a massive
number of virtualized elements has been made.
ETSI

---------------------- Page: 6 ----------------------
7 ETSI TR 103 527 V1.1.1 (2018-07)
The present document is one of three Technical Reports addressing this issue:
• ETSI TR 103 527 (the present document): "Virtualized IoT Architectures with Cloud Back-ends" (the present
document);
• ETSI TR 103 528 [i.1]: "Landscape for open source and standards for cloud native software for a Virtualized
IoT service layer";
• ETSI TR 103 529 [i.2]: "Virtualized IoT over Cloud back-ends: A Proof of Concept".

ETSI

---------------------- Page: 7 ----------------------
8 ETSI TR 103 527 V1.1.1 (2018-07)
1 Scope
The present document:
• makes a description of some use cases that benefit from virtualization and outlines which one will be used for
the Proof-of-Concept that is described in depth in ETSI TR 103 529 [i.2];
• addresses the rationale and requirements for the use of virtualization - and of the cloud in general - in support
of IoT systems. It also introduces some features that will be key for the definition and further implementation
of virtualized IoT systems such as microservices;
• provides the identification of new architectural elements (components, mappings, Application Programming
Interfaces (API), etc.) that are required to address IoT on a cloud back-end. In particular, one objective of the
present document is to describe how current IoT nodes e.g. the oneM2M CSE, can be modified and improved
by the introduction of micro-services.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or non-
specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long-term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI TR 103 528: "SmartM2M; Landscape for open source and standards for cloud native
software applicable for a Virtualized IoT service layer", 2018.
[i.2] ETSI TR 103 529: "SmartM2M; IoT over Cloud back-ends: a Proof of Concept", 2018.
[i.3] ITU-T News: "What is 'cloud-native IoT' and why does it matter?", October 2017.
NOTE: Available at http://news.itu.int/what-is-cloud-native-iot-why-does-it-matter/.
[i.4] Amazon Web Services: "What is Auto-scaling".
NOTE: Available at http://docs.aws.amazon.com/autoscaling/latest/userguide/WhatIsAutoScaling.html.
[i.5] Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the
protection of natural persons with regard to the processing of personal data and on the free
movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation).
NOTE: Available at https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=celex%3A32016R0679.
[i.6] Deloitte: "Data Privacy in the cloud", 2016.
NOTE: Available at https://www2.deloitte.com/content/dam/Deloitte/ca/Documents/risk/ca-en-risk-privacy-in-
the-cloud-pov.PDF.
[i.7] ETSI TS 118 101 (V2.10.0): "oneM2M; Functional Architecture (oneM2M TS-0001
version 2.10.0 Release 2)".
ETSI

---------------------- Page: 8 ----------------------
9 ETSI TR 103 527 V1.1.1 (2018-07)
[i.8] Recommendation ITU-T Y.3600: "Big data - Cloud computing-based requirements and
capabilities", 2015.
[i.9] ETSI GS NFV 002: "Network Functions Virtualisation (NFV); Architectural Framework".
[i.10] ETSI GS NFV-INF 001: "Network Functions Virtualisation (NFV); Infrastructure Overview".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
Open Source Software (OSS): computer software that is available in source code form
NOTE: The source code and certain other rights normally reserved for copyright holders are provided under an
open-source license that permits users to study, change, improve and at times also to distribute the
software.
source code: any collection of computer instructions written using some human-readable computer language, usually as
text
standard: output from an SSO
Standards Setting Organization (SSO): any entity whose primary activities are developing, coordinating,
promulgating, revising, amending, reissuing, interpreting or otherwise maintaining standards that address the interests
of a wide base of users outside the standards development organization
NOTE: In the present document, SSO is used equally for both Standards Setting Organization or Standards
Developing Organization (SDO).
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
AE Application Entity (in oneM2M)
AMQP Advanced Message Queuing Protocol
API Application Programming Interface
ARM Acorn RISC Machine architecture
BCP Best Common Practices
CAPEX Capital Expenditure
CEP Complex Event Processing
CoAP Constrained Application Protocol
CPU Central Processing Unit
CSC Cloud Service Customer
CSE Common Services Entity (in oneM2M)
CSF Common Service Function
CSP Cloud Service Provider
DDoS Distributed Denial of Service
EU European Union
GDPR Global Data Protection Regulation
HLA High Level Architecture
HTTP HyperText Transfer Protocol
IaaS Infrastructure as a Service
IAM Identity and Access Management
ICT Information and Communication Technology
IoT Internet of Things
IP Internet Protocol
IPC Inter-Process Communication
IPE Interworking Proxy Entity (in oneM2M)
ETSI

---------------------- Page: 9 ----------------------
10 ETSI TR 103 527 V1.1.1 (2018-07)
ISG Industry Specification Group
IT Information Technology
MANO MANagement and Organization (in NFV)
MQTT Message Queuing Telemetry Transport
NFV Network Function Virtualisation
NFVI NFV Infrastructure
ONAP Open Network Automation Platform
OSM Open Source Mano (in ETSI)
OSS Open Source Software
PaaS Platform as a Service
PoC Proof-of-Concept
PoP Point of Presence
SaaS Software as a Service
SDO Standards Development Organization
SE Service Entity (in oneM2M)
SPOF Single Point Of Failure
SSO Standards Setting Organization
UC Use Case
URI Uniform Resource Identifier
VM Virtual Machine
VNF Virtualized Network Function
4 Rationale for IoT Virtualization
4.1 IoT: towards massive deployments
The focus of IoT in the recent years has been on connecting devices and applications. To this extent, a number of
standards, frameworks, solutions have been developed. Now that the maturation of the industry is progressing rapidly,
IoT is facing to major challenges.
On the one hand, connected devices as well as applications have to be integrated with existing, evolving or entirely new
business processes: this creates the need for very adaptive frameworks that offer the possibility to easily introduce new
applications and to ensure that they are properly connected to the existing enterprise systems, and to process enormous
quantity of data.
One the other hand, IoT systems are transitioning from proof-of-concept deployments or new projects with limited size
and scope towards full-fledge systems. These new systems may require extremely high numbers of connected devices
(thus generating needs for scalability or deployment automation) as well as stringent non-functional requirements (such
as low latency).
In both cases, new IoT systems will require a high degree of availability, adaptability and flexibility. In particular, the
resources used by those systems may have to be very dynamic, both in terms of configuration and run-time flexibility.
The models provided by Cloud Computing, which have been designed upfront with these two requirements in mind,
seem very attractive in this context.
4.2 Cloud Computing and Virtualization
Cloud computing is allowing the provision of very sophisticated capabilities; for computing, storage, analytics, etc.; to
very dynamic and potentially massive number of users. Those capabilities are provided as services
(Platform-as-a-Service, Infrastructure-as-a-Service; Software-as-a-Service; etc.) that provides functional and also non-
functional support (e.g. low latency fault-tolerance, horizontal scalability, cost-optimization, or geo-optimization
together with Service Level Agreements (SLAs), and security.
The technical capabilities of cloud computing technology made it possible to provide the most demanding information
and communication technology (ICT) infrastructures, such as communication networks, from specialized hardware and
software to new software paradigms, referred to as 'cloud-native'.
ETSI

---------------------- Page: 10 ----------------------
11 ETSI TR 103 527 V1.1.1 (2018-07)

Figure 1: Options for adoption of Cloud Native solutions
The expectation of Cloud-Native applications is to benefit from offerings from Cloud Service Providers (CSP) that may
cover parts or all of the layers of Virtualized app
...