Introduction to centralized Logs

Introduction

In this Q4 we have started a series of new functionalities guided towards the log integration. The main goal si not only to bring together the platform logs in a single repository, but also to make it easy to run searches or other analysis on them.

By using Graylog, you will be able to analyze the content of your logs in realtime in a very quick and easy way.

What is Graylog?

Graylog is an open-source centralized log management solution that, among other things, provides you with standard log capture, sorting and real-time analysis of your logs.

More information at : https://www.graylog.org/products/open-source

Open

Use in Onesait Platform

Let's now describe how the platform modules are connected and how any application developed on the platform can also do so.

Connection to the platform

INPUTS:

These are the components that Graylog define as log ingestion origins. In this case, we will be using a GELF (Graylog Extended Log Format) TCP input.

Open

STREAMS:

They are a mechanism for log classification and categorization. By using rules, Graylog can classify each log entry in a Stream. In the case of the platform, each log has a number of custom fields by which we can classify its origin. The most important among these custom fields is the "app_name" field, that indicates the source module of each entry. To facilitate searches, we have created one Stream per module, so that it can be quickly selected when searching:

CUSTOM FIELDS:

As discussed above, you can define custom fields when sending logs. This will allow you to use them not only in the Streams, but also for your searches. For example, in the case of the FlowEngine, we add the "domain" field which will have at least 3 values:

• app.js: main NodeJS process manager of the different domains.

• Proxy-nodered.js: Internal Proxy used by app.js to route to each NodeRED instance

• “DomainName”: Let's imagine that you have created, with your user, a domain in the FlowEngine called "myUserDomain". If you search for that string, you can see the logs generated only by your domain:

SEARCHING:

All fields, whether custom or not, can be used in your searches, both for display and for filters:

Connecting other components

Once Graylog is running on the platform, you can take advantage of it to add logs of applications developed on it, such as microservices.

For this, you have 2 options:

Using GELF-TCP:

Same as the rest of the platform components, it is possible to connect via GELF TCP to port 12201.

Same as for the rest of the modules from Onesait Platform, the GELF TPC connection is available on port 12201. Full example here: How to create a microservice that writes to the centralized log tool

Using Sidecars:

Sidecars are lightweight configuration systems for different log collectors or backends. The Graylog node(s) act as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service (Windows host) or daemon (Linux host).

The configuration of the log collectors is centrally managed by Graylog, which distributes them via sidecar automatically. eriodically, the Sidecar daemon will fetch all relevant configurations for the target. These sidecars activate collectors (supporting FileBeat, Winlogbeat and NXLog by default) that will send the logs to Graylog.

For more information on how to configure sidecars and collectors: https://docs.graylog.org/en/4.0/pages/sidecar.html

Next steps

In Q1 of 2021, Graylog will be fully integrated with the Onesait Platform Identity Manager, and accessible via the ControlPanel together with an access management derived from the platform users. Up until then, it will be accessed from a separate URL and with a single administrator user.

Vídeo explanation (in Spanish)

https://youtu.be/qG-s-HX9S2s