Introduction

A couple of new JDK utilities have been added to the base image of the Onesait Platform containers: JStack and JMap, which by default are not included in the OpenJDK base images.

Both tools are useful when certain modules start to work in an anomalous way, either due to inappropriate use or requiring fine-tuning of parameters, according to the project where they are being used.

Some of the anomalies that these tools consider include:

• Out-Of-Memory errors.

• Modules unable to respond to new requests.

• Connection pool exhaustion errors.

• Worker pool exhaustion errors.

• Performance degradation.

What is JStack?

JStack is a tool designed to generate a thread dump of a running Java Virtual Machine (JVM). This dump shows detailed information about the threads that are running, including:

• Thread States: Such as whether they are running, blocked, or waiting.

• Stack Traces: Lines of code where each thread is located.

• Resource Locks: Threads that are waiting for access to shared resources.

Main Use Cases

In general, JStack is used in:

1. Deadlocks Diagnosis: As it identifies threads that are waiting indefinitely for resources.

2. Performance Analysis: It detects bottlenecks by observing which threads are consuming the most resources.

3. Error Resolution: It obtains context when an application is unresponsive or exhibits unusual behaviour.

How do I use JStack?

To use JStack, simply run the following command from the command line:

jstack <pid>

Where pid is the process identifier of the running JVM process. The PID can be found using tools such as jps (Java Process Status) or ps on Unix systems.

What is JMap?

JMap is another tool used to inspect the memory usage of a Java application. It provides detailed information about the JVM heap, such as:

• Heap Structure: With the distribution of objects in memory.

• Usage Statistics: With the amount of used and available memory.

• Object Histograms: With information about the type and number of objects in the heap.

Main use cases of JMap

The use of JMap is useful for:

1. Detecting Memory Leaks: By analysing the heap dump it will be possible to identify objects that are not released correctly.

2. Optimising Memory Usage: It allows to understand which parts of the application consume more resources.

3. Analysis of the Garbage Collector's Behaviour: This allows you to evaluate how objects are accumulated and eliminated in the application.

How to use JMap?

JMap has a number of useful options, including:

• Obtaining a histogram of objects:

jmap -histo <pid>

• Generate a heap dump:

jmap -dump:format=b,file=heap_dump.hprof <pid>

This file can then be analysed with tools such as Eclipse MAT (Memory Analyzer Tool).

How to use JStack and JMap with Onesait Platform modules

Both tools are used in a similar way. The following will illustrate the process using both in an OpenShift environment with Onesait Platform deployed.

First, connect via OpenShift to the terminal of the Pod to which you want to perform a thread or memory dump.

The PID of the process shall then be identified using the following command:

ps –ef | grep java

After execution, the PID of the process will be identified as 1. Knowing this, you will be able to do a thread and/or memory dump. This will be done with the following commands:

jstack 1 > /tmp/volcadohilos.txt

This will generate the thread dump in the /tmp/threaddump.txt file.

jmap -dump:format=b,file=/tmp/volcadomemoria.hprof 1

This will result in the memory dump in the /tmp/memorydump.hprof file.

Once these files have been generated, they must be downloaded from the container to the local machine, where they will be analysed. To do this, use these two utilities:

• oc: Openshift CLI (Command Line Interface).

• kubectl: Generic CLI for Kubernetes.

With oc, the machine will be authenticated in the OpenShift (Kubernetes) cluster and will be able to access the Pod where the thread and memory dumps have been generated to download them.

To do this, the user options will be displayed in the Openshift web console:

From the different options that appear, select ‘Copy login command’. This will cause you to re-authenticate to the web console, and you will be redirected to a blank screen where only the e will have the ‘Display Token’ option.

Clicking on the link will display the oc login command to be executed:

Copy the entire oc login command and run it in a terminal on your computer:

Next, select the project where the Pod you want to access is located using the following command:

oc project <project_name_openshift>

Finally, the files will be copied to the local machine with the following command:

kubectl cp <pod_name>:<pod_file_path> <local_file_path>

In the case of the Platform:

kubectl cp controlpanelservice-b-7b47bff6c8-64h9d:/tmp/volcadohilos.txt volcadohilos.txt
kubectl cp controlpanelservice-b-7b47bff6c8-64h9d:/tmp/volcadomemoria.hprof volcadomemoria.hprof

Analysing thread dumps with JStack

There are a multitude of tools for analysing a thread dump. Some of them are:

• Eclipse or IntelliJ, through plugins.

• VisualVM.

• IBM's TDA.

• FastThread.

In the case of Onesait Platform, FastThread will be used, as it is a web service that does not need to be installed on the computer and which allows the file generated with JStack to be uploaded and analysed very efficiently, even applying algorithms to detect problems.

Once the thread dump has been loaded into the service, the following control panel will be displayed, where the threads have been automatically grouped according to the Thread Pool they belong to, and their status. This will give a quick overview of the status of the module when the thread dump was extracted:

From this view, knowing that in this particular case the problem was the exhaustion of the pool of connections to the database, it will be possible to focus on two points:

• Blocked Threads:

• Thread Pools that are at Maximum Capacity:

Reference can be made to the source code to identify the business logic that is causing the problem, which helps to detect that, in this particular case, an inappropriate number of asynchronous tasks, which were then waiting for a change of state, were reserving the database connection during the entire wait for the event change.

Thread dump analysis with JMap

There are also a multitude of tools for analysing a memory dump, such as:

• Eclipse MAT.

• VisualVM.

• JProfiler.

For the case of the Platform, we are going to talk about Eclipse MAT, which is an eclipse environment designed exclusively for analysing memory dumps.

When loading a memory dump, you can see a small control panel showing the distribution of memory usage, along with possible problems encountered, derived from objects that consume a fairly high percentage amount of memory relative to the rest:

Locating memory problems usually relies on browsing the Dominator Tree of objects, sorted by Eclipse MAT according to their size:

It is also possible to examine the histogram, which shows a summary of all objects in the heap, organised by type and provides key information on their distribution and use.

Conclusion

Having these tools in the base image of the Onesait Platform container will make it possible to generate evidence of problems that can be analysed by the support team, in order to determine the necessary configuration according to each project's needs, as well as to detect and correct bugs.