Introduction

In this release, the connection functionality to different Kafka clusters has been integrated to manage the most common components of a cluster, such as topics and consumer groups.

The basic functionalities will be the following ones:

• Connections:

  • Create a connection to a cluster, setting the necessary connection properties.

  • Modify / delete connections.

  • List the connections.

  • Access the data of a Cluster: Topics and Consumer Groups.

• Topics:

  • Creation / modification / deletion of topics: we can specify partitions, replicas and any necessary properties.

  • Other operations:

    • Data insertion.

    • Query of the last x records.

    • Data purge.

• Consumer groups:

  • Viewing consumers and sharing partitions.

  • Viewing of the state of consumption by partition and its lag.

  • Offset alteration:

    • at the beginning of the topic.

    • at the end of the topic.

    • at a certain timestamp.

How does it work?

From the Control Panel, we can access it from the TOOL menu.mDEVELOPMENT > Kafka cluster management,

Open

where we will be shown the list of connections that we have defined.

Connections to Kafka

Open

Pressing the + button, we can create a new connection, where we will have to indicate name, description and list of brokers (host:port separated by commas)

and optionally we can indicate the properties that are necessary. Each one of them has the description in the tooltip (when you leave the cursor on it). When selecting one of them, the default value of said property will be loaded:

Same applies to modifying the connection. Any value except the identifier can be changed.

Once a connection has been created, we can see its status and elements by clicking on the “view” icon.

We can see three sections:

• General Information: Brokers that make up the cluster.

• Topics:

• Consumer groups:

Topics

Once we are inside a connection, we can perform the following operations:

• View a topic:
  
  

  
  It allows us to see the properties defined manually or by default,
  
  

  
  as well as the partitions.
  
  

• Create/Modify Topic.

• Delete topic:
  
  

  
  
  

• Consume/produce data:
  
  

   

  • Consume data: We can consume the desired number of records from the beginning of the topic, from the end (new insertions), or from a given moment. We will indicate the maximum number of records to read, as well as the time for which we are going to read. The platform will launch a consumer until the indicated time has elapsed, or until the number of records indicated in the form has been read. A timeout greater than 4 seconds is recommended, to give the client time to boot up and access the data.
    
    

     

  • Produce data: It allows us to manually insert data to the topic.
    
    

     

  • Purge data: It will mark as deleted the data that the topic had.
    
    

    
    
    

Consumer groups

Within a connection, we can view the list of consumer groups. We can perform the following operations on them:

• View status: We can see the status of each client within the group, as well as the distribution of their partitions.
  
  

  
  We can also see the group offsets on each partition.
  
  
  

• Edit offsets: It allows us to change the offsets of a group of consumers, as long as it is not active.
  
  

  
  Once in the editing screen, we can select between moving the offsets to the beginning, end or a certain moment for a topic and all its partitions:
  
  

   

• Delete Group: This option only deletes the group of consumers if it is in “Empty” state.