How to configure distributed tracing in Java Microservices?

Owned by Onesait Platform
Last updated: 13/07/23
5 min read

Available from version 5.1.0 of Onesait Platform (Survivor).

Introduction

In the Q2 2023 release, we have included the capability of distributed Tracing in the Platform.

This allows to get distributed traceability between its components and other components like microservices that can call a platform REST API and be able to see this full traceability from within this microservice.

How to access the functionality

These traces can be seen from the Control Panel in the TRACING option:

This example shows the traces of a microservice that calls another microservice, and this in turn calls a REST service exposed from the Platform’s API Manager. In the traces you can see how much time has been spent in each span, which is useful to detect bottlenecks, interruptions between components,...

How to configure your microservice to trace?

Microservice instrumentation

The instrumentation of yourmicroservice will allow it to generate trace and span information that it will send to the collector. These traces and spans share a context so that, from the beginning to the end, the path traveled, the time at each point of the path, and the time can all be traced. It also contains information of the key-value type, to later be able to interpret everything and be able to draw conclusions.

The Platform uses Open Telemetry to create the traces and spans and send it to the collector for processing.

Instrumentation Types

There are 2 types of instrumentation:

  • There is the automatic instrumentation. It consists of using the Open Telemetry agent to, in a non-intrusive way, obtain the traces without having to touch the code of the microservice, application, etc., from which the traces are intended to be obtained, since the agent is in charge of listening to the different, most important libraries that are usually used, and from these it generates the traces. It is the one we will use by default.

  • Then there is also the manual instrumentation. In this case, you have to add the libraries with the Open Telemetry SDK to the application, microservice, and implement how the different traces are going to be generated. This case is more useful when it is intended to generate certain types of traces with specific or more customized information.

Instrumentation Example

For the examples, you will create two Spring Boot microservices, one in which you will configure the automatic instrumentation, and another for the manual one.

Automatic instrumentation

For this type of instrumentation, all you have to do is start the Open Telemetry agent together with the application, which is highly configurable by parameters.

In the simplest example, you only have to specify where the jar is located in the JVM parameters:

-javaagent:/path/opentelemetry-javaagent.jar

and add the following environment variables:

OTEL_JAVAAGENT_ENABLED=false #(Dis)Enables el agente OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger-service:4317 #collector endpoint OTEL_SERVICE_NAME=microservice #service name OTEL_METRICS_EXPORTER=none #So that it does not export metrics, since this feature is not activated

The agent can be downloaded with this url:

download java agent. (opentelemetry-javaagent-1.27.0.jar)

There are also agents for other technologies.

Manual instrumentation

In this case, you have to add code to the application to generate the traces, adding these dependencies to the pom.xml file:

<dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api</artifactId> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-sdk</artifactId> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-otlp</artifactId> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-semconv</artifactId> <version>1.27.0-alpha</version> </dependency> </dependencies> <dependencyManagement> <dependencies> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-bom</artifactId> <version>1.27.0</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement>

and define some information in the application.properties file:

otel.config.trace-id-ratio-based: 1.0 otel.exporter.otlp.endpoint: https://lab.onesaitplatform.com/otelcollector/ service.name: Onesait-Platform-Microservice

In this case, configure the base ratio, the url where the Otel collector is located, and the name of the service.

Finally, in the microservice, what is done is to invoke a REST service generated with the platform API Manager, so the purpose is to create a trace that wraps this call.

@Autowired private TracingConfiguration tracingConfiguration; private static final Logger log = LoggerFactory.getLogger(ExampleRestController.class); private Tracer tracer; @PostConstruct public void init() { tracer = tracingConfiguration.getOpenTelemetry().getTracer(ExampleRestController.class.getPackageName()); } @Autowired HttpServletRequest request; //@RequestHeader(required = false) HttpHeaders headers @GetMapping(value = "/getAll") public void getAll() throws IOException { request.getHeaderNames(); Context extractedContext = tracingConfiguration.getOpenTelemetry().getPropagators().getTextMapPropagator() .extract(Context.current(), request, Utils.getter); // the span is created try (Scope scope = extractedContext.makeCurrent()) { Span span = tracer.spanBuilder("example span with context propagation").setSpanKind(SpanKind.SERVER) .startSpan(); try (Scope ignored = span.makeCurrent()) { URL url = new URL("https://lab.onesaitplatform.com/api-manager/server/api/v1/customer/"); // from here you would be making a call to a platform rest api that queries the // information of an entity for example HttpURLConnection con = (HttpURLConnection) url.openConnection(); // adding information to the span is optional but recommended span.setAttribute(SemanticAttributes.HTTP_URL, url.toString()); span.setAttribute(SemanticAttributes.HTTP_METHOD, "GET"); span.setAttribute("component", "http"); // Inject the request with the *current* Context, which contains our current // Span. tracingConfiguration.getOpenTelemetry().getPropagators().getTextMapPropagator() .inject(Context.current(), con, Utils.setter); // the necessary information is added to the request con.setRequestProperty("accept", "application/json"); con.setRequestProperty("X-OP-APIKey", "6bfb064cb62548b78d7...."); con.setRequestMethod("GET"); int status = con.getResponseCode(); log.info("status: " + status); } finally { // the span is closed span.end(); } } }

 

In this example, the trace is obtained:

@PostConstruct public void init() { tracer = tracingConfiguration.getOpenTelemetry().getTracer(ExampleRestController.class.getPackageName()); }

Then the context is obtained, in case this microservice was invoked from another one that had already created a previous trace. With this, we achieve that the trace is not divided into fragments that start from different services, and that all start from an initial call.

Context extractedContext = tracingConfiguration.getOpenTelemetry().getPropagators().getTextMapPropagator() .extract(Context.current(), request, Utils.getter);

Then, within the context, a span is created with specific information that the developer wants to be shown in the traces.

try (Scope scope = extractedContext.makeCurrent()) { Span span = tracer.spanBuilder("example span with context propagation").setSpanKind(SpanKind.SERVER) .startSpan();

When invoking the REST service, the current context information is added, so that the next component of the trace has the previous context and can decide whether to create a new trace or continue with the current one. This is achieved with the propagators.

tracingConfiguration.getOpenTelemetry().getPropagators().getTextMapPropagator() .inject(Context.current(), con, Utils.setter);

Finally, it is always advisable to close the span.

} finally { // the span is closed span.end(); }

If we go to the traces and expand the detail, you can check that the specific information added by the developer is visible.

 

To verify that the context is not lost with this implementation, we created another microservice, Onesait-Platform-Microservice2, identical to the one shown but that called Onesait-Platform-Microservice instead of invoking the platform REST service, which is it shows in the first trace the service Onesait-Platform-Microservice2 → Onesait-Platform-Microservice → api-manager.