Introduction

In this article we are going to see how an external application can be configured so that its execution metrics are stored and displayed with the monitoring tool used by Onesait Platform.

As a requirement to be able to use the functionalities described in this guide, you will need to have the Onesait Platform monitoring module deployed and correctly configured.

Tools

Onesait Platform monitoring is based on two main technologies, Prometheus and Grafana.

Prometheus is a widely used timeseries database for monitoring metrics.

On the other hand, Grafana is a tool that allows you to create dashboards by configuration, and in the same way as Prometheus, it is widely used in monitoring systems and software, so it has a good base of dashboards ready for use and created by its community.

Besides, since Onesait Platform is deployed with Kubernetes, Prometheus and Grafana Operator are used, making it easy to deploy and configure Prometheus and Grafana on Kubernetes.

Bear in mind that the purpose of this monitoring tool is to have metrics of the software deployed in the Kubernetes cluster, by default, the metrics of the Onesait Platform modules, and as we will see in this article, also the metrics of other applications. This does not replace the hardware or network monitoring that exists in an installation.

Application Settings

In this article we are going to see a typical configuration case for external applications. Specifically, we will see how to store the metrics in Prometheus and how to display them in Grafana, considering that the Onesait Platform monitoring module is being used, which makes these services available and configured.

Prometheus gets metrics from apps directly, so apps need to enable a URL where they can query the metrics. This action is known in Prometheus as "scrape". If a third party application does not enable the metrics in a format that Prometheus can use, there are several alternatives to incorporate them. The easiest one is to deploy an agent, which in Prometheus terminology is called an exporter, and which transforms the metrics from the format used by the application to a format accessible to Prometheus.

In this case we are going to do an example with a Spring-Boot application. Specifically, the Petclinic sample application that anyone can obtain from https://github.com/spring-projects/spring-petclinic will be used.

In the case of Spring-Boot, it does natively support enabling an actuator with the metrics generated for Prometheus. By default, this project already has the actuators configured. In Spring-boot, actuators enable endpoints with information and commands for application operation. If you do not have them enabled, you must add the following dependency to the project's pom.xml file:

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

For more information about actuators in Spring-boot, check the official documentation: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.enabling

In Spring-Boot 2.x, this dependency alone is not enough to have metrics in the format expected by Prometheus, since the actuator for Prometheus is not part of the default enabled actuators and requires an additional dependency.

    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

Once these changes have been added, simply start the application and test that the Spring-Boot actuator has been created correctly. You can easily test this by making an HTTP GET request to the URL http://localhost:8080/actuator/prometheus, assuming the application is running on the local machine and on port 8080. You can use either a browser or a tool like curl or Postman.

The result will be something similar to the following text:

# HELP application_started_time_seconds Time taken (ms) to start the application
# TYPE application_started_time_seconds gauge
application_started_time_seconds{main_application_class="org.springframework.samples.petclinic.PetClinicApplication",} 6.177
# HELP logback_events_total Number of events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{level="warn",} 0.0
logback_events_total{level="debug",} 0.0
logback_events_total{level="error",} 0.0
logback_events_total{level="trace",} 0.0
logback_events_total{level="info",} 18.0
...

In the case of metrics, we recommended not to configure security and limit network access so that obtaining the metrics has the least possible impact on application performance. In the case of Kubernentes, these are typically only accessible from within the cluster, and, if it is necessary to further limit access, you can use Network Policies.

Create a Docker image of our project

For this example, you are going to package our sample application in a Docker container that we will use to deploy to Kubernentes. Each project will use its own configurations to package the code. In this section, you are going to create a basic configuration so that the example can be followed with the Petclinic project.

Download the source code of the application:

git clone https://github.com/spring-projects/spring-petclinic.git

First of all you have to generate the jar with the application. To do this, run in the project’s root:

mvn clean package

This should create a jar file in the target directory of the project.

To create the docker image, you will use the following Dockerfile created in the root of the project.

FROM openjdk:11.0.1-jre-slim-stretch
EXPOSE 8080
ARG JAR=spring-petclinic-*.jar
COPY target/$JAR /app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

To make this tutorial easier to follow, you can use the onesaitplatform/example-monitoring-spring-boot:v1.0 Docker image that is hosted on DockerHub for the rest of the example.

Prometheus configuration

Once your application is correctly configured to export metrics, you have to configure Prometheus to obtain the metrics.

Since Prometheus Operator is being used, all the configuration is done with yaml files, like any other Kubernetes cluster resource, which makes it easy for this configuration to be included in any automated deployment process or in a Helm chart.

There are several ways to tell Prometheus where to look for metrics, but in most cases configuring an element called ServiceMonitor will suffice. With this component, you will tell Prometheus the necessary information to obtain the metrics.

Below is the ServiceMonitor for your sample application.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-monitoring
  namespace: examples
spec:
  endpoints:
  - path: /actuator/prometheus
    targetPort: 8080
  namespaceSelector:
    matchNames:
    - examples
  selector:
    matchLabels:
      application: app-for-monitoring
  targetLabels:
  - application

In this case you are configuring a Service Monitor in the examples namespace, which is where you have deployed the example application, to group the resources in the same namespace.

The next thing to configure is the endpoint where Prometheus can get the metrics, which in this case is in the path /actuator/prometheus and in port 8080, which is the one that you are exposing within the cluster in the service of your example application.

To identify the service from which to obtain metrics, you can filter by namespace, examples in this case, and by labels. Keep in mind that you are selecting a service, so the labels must be configured in the service and they do not need to be in the deployments. Prometheus will fetch metrics from all pods pointed to by Prometheus with no additional configuration required.

For more information on how to configure a ServiceMonitor, check: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitor

and for full information on Prometheus Operator: https://github.com/prometheus-operator/prometheus-operator/tree/main/Documentation

To deploy a Service Monitor in Rancher, you must do it in Cluster Explorer → Monitoring (monitoring must be previously configured):

Then choose the Service Monitors option and create a new Service Monitor using the yaml file.

Once this component has been created, and after a minute or two, you can check if Prometheus has been able to obtain metrics from our application. The first place to check will be the Prometheus Targets. These can be seen in Cluster Explorer → Monitoring → Overview → Prometheus Targets.

In the list of connected services, you can see the one you just created and its status.

Once it is in UP state, Prometheus will already be collecting metrics from your program.

You can check them using a query from the option Cluster Explorer -> Monitoring -> Overview -> Prometheus Graph.

By default, Prometheus will collect metrics every minute and they will be retained in Prometheus for a limited time, depending on how monitoring has been configured in the particular environment. Obviously, greater retention time implies greater amount of disk.

Grafana configuration

Once you have the metrics stored, you have to be able to view them. Although they can be seen through queries directly to Prometheus, it is normal to define dashboards that show us the most relevant metrics in which you are interested.

Grafana is configured by default without persistence, so that all the configuration is in Kubernetes ConfigMaps. This has several advantages, but the main one is that the configuration of the dashboards themselves can be done in the same application deployment using, for example, Helm.

In the case of Spring-Boot, there is already a dashboard that shows the metrics for these applications.

If you want to create a new dashboard, you can do it very easily by creating a new dashboard in the normal way in Grafana, either from scratch or by importing an existing one. Once you consider that the dashboard is considered, simply export it as JSON and create a yml to create a ConfigMap using the json:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    grafana_dashboard: "1"
  name: name-of-your-dashboard
  namespace: cattle-dashboards # Change if using a non-default namespace
data:
  name-of-your-dashboard.json: |-
    {
    ...
    }

Grafana will automatically create the dashboard without needing to restart anything. It is important that the grafana_dashboard label has a value of 1, and that the namespace is the one that the Grafana Operator has configured. In the case of Rancher, this namespace is cattle-dashboards.