Introduction

The Digital Broker simplifies communication with the Platform by isolating itself from the underlying communication protocol.

Out-of-the-box, it offers the following protocols:

• REST

• Kafka

• MQTT

• Websocket

• Video RTSP

These protocols define a messaging protocol that manages the communication between the client and the platform. It allows bidirectional communication, subscriptions, etc.

In addition, the Digital Broker allows the incorporation of new protocols through the plugin concept.

Protocols supported

REST

REST is an interface for connecting various systems based on the HTTP protocol (one of the oldest protocols) and is useful to obtain and generate data and operations, returning that data in very specific formats, such as XML and JSON.

The most used format nowadays is the JSON format, as it is lighter and more readable compared to the XML format. Choosing one will depend on the logic and needs of each project.

Kafka

Introduction

The following are a few points that describe Apache Kafka:

• Kafka is a distributed event store and Stream event processing platform.

• Kafka is open source and written in Java.

• The main goal in designing Kafka is to handle real-time data and provide a high-throughput, low-latency platform.

• Kafka is a Stream event platform that has many capabilities to publish (write) and subscribe (read) to stream events from a different system.

• In addition, to store and process events in a long-lasting way, for as long as we want, by default, Kafka stores events in a secure way.

• Kafka is a distributed system, which has servers and clients communicating via TCP.

• It can be deployed in different virtual machines and containers in local and cloud environments as needed.

• In the Kafka world, a producer sends messages to the Kafka Broker. Messages are stored within topics and the consumer subscribes to that topic to consume the messages sent by the producer.

• ZooKeeper is used to manage the metadata related to Kafka. It keeps track of the brokers that are part of the Kafka cluster and the partitions of the different topics. Finally, it manages the status of Kafka nodes and maintains a list of Kafka topics and messages.

Open

Kafka's main concepts

Open

• Event: An event or log is the message that we read and write to the Kafka server; we do this in the form of events, and it contains a key, a value, a timestamp and other metadata headers.

• Producer is a client application that sends messages to the Kafka node or broker.

• Consumer is an application that receives data from Kafka.

• Kafka cluster is the set of Kafka processes that share the workload with each other for different purposes.

• Broker is a Kafka server that acts as an agent between the producer and the consumer, which communicate through the broker.

• Topic: Events are stored inside the "topic" (in the style of a file folder). Each topic has one or more producers and consumers, which write and read data from the topic. Events in the topic can be read as many times as necessary because it persists events (Kafka is not like other messaging system that deletes messages after consuming them).

• Partitioning: Topics are partitioned, which means that the topic is spread over multiple partitions that we create within the topic. When the producer sends an event to the topic, it will store it within the particular partitions, and then, the consumer can read the event from the corresponding topic partition in sequence.

• Offset: Kafka assigns a unique ID to a message stored within the topic partition when the message arrives from the producer.

• Consumer groups: In the Kafka world, the consumer group acts as a single logical unit.

• Replication: In Kafka, to make data fault tolerant and highly available, we can replicate topics in different regions and brokers.

Kafka APIs

Kafka has 5 basic APIs that serve different purposes:

• Admin API: This API manages the different Kafka topics, brokers and objects.

• Producer API: This API is used to write/publish events in different Kafka topics.

• Consumer API: This API is used to receive the different messages corresponding to the topics subscribed by the consumer.

• Kafka Stream API: This API is used to perform different types of operations such as time windows, joins, aggregation and many others.

• Kafka Connect API: This API works as a connector to Kafka, which helps different systems to connect to Kafka easily. It has different types of ready-to-use connectors related to Kafka.

MQTT

Message Queue Telemetry Transport (MQTT) is an open Machine-to-Machine (M2M) connectivity protocol that allows telemetry-style data to be sent as messages over high latency or constrained networks.

MQTT is a connectivity protocol focused on M2M (machine-to-machine) and IoT (Internet of Things) as it is designed to be an extremely lightweight TCP-based messaging protocol.

It is useful for connections to remote sites where a small foodprint (such as Arduino) is essential and bandwidth is very important.

A very important feature is that, being such a lightweight protocol, there are MQTT clients and servers in several languages (http://mqtt.org/software) , from Java and C to Arduino, and JavaScript. In addition, many messaging brokers such as IBM MQ or RabbitMQ support this lightweight protocol, so that the client can publish and subscribe to this messaging through this protocol.

WebSockets

WebSocket is a TCP-based network protocol that establishes how data should be exchanged between networks. Because it is a reliable and efficient protocol, it is used by virtually all clients. The TCP protocol establishes connections between two communication endpoints, called sockets. In this way, data can be exchanged in both directions.

In bidirectional connections, such as those created by WebSocket (sometimes also websocket or web socket), data is exchanged in both directions at the same time.

The main features of WebSockets are:

• Bidirectional: Both the client and the Server can initiate connections.

• Full-duplex.

• Single TCP connection: With REST a new TCP connection is required for each request, whereas with WebSockets the client and server communicate on the same request.

• Lightweight protocol: Compared to an HTTP connection, in WebSockets, once the handshake is done, the data is minimized.