/
How to write to a Kafka topic and consume from a Notebook

How to write to a Kafka topic and consume from a Notebook

13/09/21

In this tutorial, we will create a flow that publishes information in a Kafka topic in the platform, later recovering the data and painting them in a Notebook.


Creation of the intake pipeline in Kafka

We will start by creating a pipeline. From a user with analytics/datascientist role, we will go to the Analytics Tools → My pipelines menu and click on the "Create Pipeline" button and give it a name, for example "KafkaTest".

Once the pipeline is created, we will add a destination node, type "Kafka Producer", and configure it as follows:

  • URI Broker: <machine>: <port>
  • Topic: Name of the Kafka topic where we will publish the data. If the topic does not exist, then it will be created by default.

To keep the example simple, we will generate random measurement data for a device over time.

To do this we will create a "DEV Data Generator" node that generates random numerical values ​​as follows:


We will configure in the same node the record generation periodicity to one every 5 seconds:


Lastly, we will create a "JavaScript Evaluator" node to add some non-random values ​​such as a timestamp representing the moment, and to always adjust the record's id to the same device:

With this pipeline, once launched, a message will be inserted every 5 seconds in the specified Kafka topic. For now, we will leave it active to generate data:



Notebook creation to represent the data sent to the Kafka topic

  1. To create the Notebook, there is an ANALYTICS TOOLS Menu option including set of tools.



  2. We select the optoin "My Notebooks" and from here we click on New Notebook and give the Notebook a name.



  3. We will represent the data in a graph with the Python interpreter by following the next steps:

    1. We import the necessary modules.



    2. We configure the Kafka consumer. The parameters are:



      1. Kafka topic that we will consume.
      2. Consumer group.
      3. Servers.


    3. We generate a dataframe to insert the data.



    4. We collect all the messages from the topic, transform the received data then paint them in a graphic:



    5. If we launch the paragraph of the Notebook, we can see the messages obtained from the topic, as well as a graphic representation of the measurements: