ETSI TR 103 528 V1.1.1 (2018-08)

ETSI TR 103 528 V1.1.1 (2018-08)






TECHNICAL REPORT
SmartM2M;
Landscape for open source and standards for cloud native
software applicable for a Virtualized IoT service layer

---------------------- Page: 1 ----------------------
2 ETSI TR 103 528 V1.1.1 (2018-08)



Reference
DTR/SmartM2M-103528
Keywords
cloud, IoT, open source, virtualisation

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2018.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TR 103 528 V1.1.1 (2018-08)
Contents
Intellectual Property Rights . 8
Foreword. 8
Modal verbs terminology . 8
Introduction . 8
1 Scope . 10
2 References . 10
2.1 Normative references . 10
2.2 Informative references . 10
3 Definitions and abbreviations . 11
3.1 Definitions . 11
3.2 Abbreviations . 11
4 A Landscape for Open Source and Standards . 13
4.1 Introduction. 13
4.2 Open Source Software, Cloud-Native Computing, IoT . 14
4.3 Content of the present document . 14
5 Open Source support to IoT Virtualization . 15
5.1 An Architecture for OSS component classification . 15
5.2 A map of Cloud-Native Software . 15
5.3 Open Source Software in support of IoT Virtualization . 16
5.3.1 The approach taken . 16
5.3.2 The role of Open Source Software eco-systems . 17
5.3.3 How to read the map . 18
6 Open Source Components for IoT Virtualization . 18
6.1 Cloud Infrastructure . 18
6.1.1 OpenStack . 18
TM
6.1.2 Amazon Web Services (AWS) . 20
TM
6.1.3 Microsoft Azure . 23
TM
6.1.4 IBM BlueMix . 25
6.2 Container . 26
6.2.1 Docker . 26
6.2.2 Rocket . 28
6.2.3 Comparison of Container Software . 29
6.3 Orchestration. 29
6.3.1 Kubernetes . 29
6.3.2 Mesos . 31
6.3.3 Zookeeper . 32
6.3.4 Docker Swarm . 33
6.3.5 Yarn . 35
6.3.6 Comparison of Orchestration Software . 37
6.4 Common Services . 37
6.4.1 Data Collection . 37
6.4.1.1 Fluentd . 37
6.4.1.2 Logstash . 39
6.4.1.3 Beats . 40
6.4.1.4 Comparison of Data Collection Software . 42
6.4.2 Communication . 42
6.4.2.1 Kafka . 42
TM
6.4.2.2 Amazon Kinesis . 43
6.4.2.3 Flume . 45
6.4.2.4 Redis . 46
6.4.2.5 Comparison of Communication Software . 47
6.4.3 Computation . 47
ETSI

---------------------- Page: 3 ----------------------
4 ETSI TR 103 528 V1.1.1 (2018-08)
6.4.3.1 Apache Flink . 47
6.4.3.2 Apache Spark. 49
6.4.3.3 Apache Storm . 50
6.4.3.4 Apache Hadoop . 51
6.4.4 Storage . 52
6.4.4.1 Apache Cassandra . 52
6.4.4.2 Apache Hive . 53
6.4.4.3 Couchbase . 55
6.4.4.4 Apache HBase . 56
6.4.4.5 Vitess . 58
6.4.5 Search Engine. 60
6.4.5.1 Elasticsearch . 60
6.4.5.2 Solr . 61
6.4.5.3 Lucene . 62
6.4.5.4 Comparison of Search Engine Software . 63
6.4.6 Data Usage . 63
6.4.6.1 Kibana . 63
6.4.6.2 Grafana . 64
6.4.6.3 Comparison of Visualization Software . 66
6.5 Monitoring . 66
6.5.1 Prometheus . 66
6.5.2 Netdata . 67
6.5.3 Comparison of Monitoring Software . 69
7 Standards support to IoT Virtualization . 69
7.1 Introduction. 69
7.2 Standards Landscapes for IoT Virtualization . 70
7.2.1 An initial list of IoT Standards from AIOTI . 70
7.2.2 A landscape of Cloud Computing Standards . 70
7.3 Recent advances in IoT Standardization . 71
7.3.1 Introduction . 71
7.3.2 Big Data . 71
7.3.3 Semantic Interoperability . 72
7.4 Advances from IoT Research. 73
8 Conclusions . 74
8.1 Assessment and Lessons Learned . 74
8.2 Guidelines and Recommendations . 75
8.2.1 Guidelines to designers and developers . 75
8.2.2 Recommendation to oneM2M . 75
8.2.3 Recommendation to AIOTI and the IoT community . 75
Annex A: Change History . 76
History . 77



ETSI

---------------------- Page: 4 ----------------------
5 ETSI TR 103 528 V1.1.1 (2018-08)
List of figures
Figure 1: The potential of Cloud-Native Infrastructures . 14
Figure 2: An HLA for IoT Virtualization . 15
Figure 3: The CNCF landscape of Cloud-Native Software Components . 16
Figure 4: A global map of OSS Components for IoT Virtualization . 17
Figure 5: The example of the Apache Hadoop ecosystem . 17
Figure 6: OpenStack architecture . 19
Figure 7: Amazon Web Services Architecture . 20
Figure 8: Microsoft Azure Architecture . 24
Figure 9: IBM Bluemix Architecture . 26
Figure 10: Docker Architecture . 27
Figure 11: Rocket Architecture . 28
Figure 12: Kubernetes architecture . 30
Figure 13: Mesos Architecture . 32
Figure 14: Zookeeper Architecture . 33
Figure 15: Docker Swarm Architecture . 34
Figure 16: Yarn Architecture . 36
Figure 17: Fluentd Architecture . 38
Figure 18: Logstash Architecture . 40
Figure 19: Beats Architecture . 41
Figure 20: Kafka Architecture . 43
Figure 21: Amazon Kinesis High-Level Architecture . 44
Figure 22: Flume Architecture . 45
Figure 23: Redis Architecture. 47
Figure 24: Flink Architecture . 48
Figure 25: Spark Architecture . 49
Figure 26: Storm Architecture . 50
Figure 27: Hadoop Architecture . 51
Figure 28: Cassandra Architecture . 52
Figure 29: Apache Hive Architecture . 53
Figure 30: Couchebase Architecture . 55
Figure 31: Apache HBase Architecture . 57
Figure 32: Vitess Architecture . 58
Figure 33: Elastic Search cluster . 60
Figure 34: Kibana interface . 64
Figure 35: Grafana dashboard . 65
Figure 36: Prometheus Architecture . 66
ETSI

---------------------- Page: 5 ----------------------
6 ETSI TR 103 528 V1.1.1 (2018-08)
Figure 37: Netdata High-level features and Architecture . 68
Figure 38: Five patterns of interoperability . 73


ETSI

---------------------- Page: 6 ----------------------
7 ETSI TR 103 528 V1.1.1 (2018-08)
List of tables
Table 1: Comparison of Container Software . 29
Table 2: Comparison of Orchestration Software . 37
Table 3: Comparison of Data Collection Software . 42
Table 4: Comparison of search Engine Software . 63
Table 5: Comparison of Data Usage Software. 66
Table 6: Comparison of Monitoring software. 69


ETSI

---------------------- Page: 7 ----------------------
8 ETSI TR 103 528 V1.1.1 (2018-08)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Smart Machine-to-Machine
communications (SmartM2M).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
In addition to interoperability and security that are two recognized key enablers to the development of large IoT
systems, a new one is emerging as another key condition of success: virtualization. The deployment of IoT systems will
occur not just within closed and secure administrative domains but also over architectures that support the dynamic
usage of resources that are provided by virtualization techniques over cloud back-ends.
This new challenge for IoT requires that the elements of an IoT system can work in a fully interoperable, secure and
dynamically configurable manner with other elements (devices, gateways, storage, etc.) that are deployed in different
operational and contractual conditions. To this extent, the current architectures of IoT will have to be aligned with those
that support the deployment of cloud-based systems (private, public, etc.).
Moreover, these architectures will have to support very diverse and often stringent non-functional requirements such as
scalability, reliability, fault tolerance, massive data, security. This will require very flexible architectures for the
elements (e.g. the application servers) that will support the virtualized IoT services, as well as very efficient and highly
modular implementations that will make a massive usage of Open Source components.
These architectures and these implementations form a new approach to IoT systems and the solutions that this STF will
investigate will also have to be validated: to this extent, a Proof-of-Concept implementation involving a massive
number of virtualized elements will be made.
ETSI

---------------------- Page: 8 ----------------------
9 ETSI TR 103 528 V1.1.1 (2018-08)
The present document is one of three Technical Reports addressing this issue:
• ETSI TR 103 527 [i.1]: "Virtualized IoT Architectures with Cloud Back-ends".
• ETSI TR 103 528 (the present document): "Landscape for open source and standards for cloud native software
for a Virtualized IoT service layer".
• ETSI TR 103 529 [i.2]: "Virtualized IoT over Cloud back-ends: A Proof of Concept".

ETSI

---------------------- Page: 9 ----------------------
10 ETSI TR 103 528 V1.1.1 (2018-08)
1 Scope
The present document:
• Recalls the main elements of the High-Level Architecture (HLA) in support of IoT Virtualization as it is
described in ETSI TR 103 527 [i.1] and how Open Source Software (OSS) and Standards can be used in the
implementation of virtualized IoT systems.
• Presents, for each of the layers (and sub-layers) of the HLA, several of the OSS components that have been
developed by the open source communities.
• Presents on-going developments in standardization that can be used in support of such implementations.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long-term validity.
The following referenced documents are not necessary for the application of the present document
...