Integrating Message Queuing Telemetry Transport (MQTT)with Kafka Connect for Processing IOT data

Abstract

The Internet of Things (IoT) architecture is defined as a layered structure in which each layer represents a coherent set of services. For supporting the communication among the different IoT entities many different communication protocols are now available in practice. For practitioners, it is often not clear which communication protocol is suitable for the various conditions in which the IoT systems need to be operated. The backbone of Internet of Things (IoT) is the communication protocols which seamlessly integrate thousands of nodes and enable a light weight data transfer process. This research is to analyze the efficiency and applicability of different Machine to Machine (M2M) protocols that are available for IoT communication. This thesis aims at exploring the capabilities of such middleware and how they can be integrated in real world application need to aggregate data on a large scale. MQTT and Kafka are two complementary technologies. Together they allow to build IoT end to end integration from the edge to the data center. Kafka Connect is a part of Apache Kafka and provides a scalable and reliable way to move data between Kafka and other datastores.