In this Q4 we have started a series of new functionalities guided towards the log integration. the main objective it’s not only to keep logs in one repository, but also to make it easy to run searches or other analysis on them.
By using Graylog we are going to be able to analyze the content of our logs in realtime in a very easy way.
What is Graylog?
Graylog is a centralized log management solution for capturing, storing, and enabling real-time analysis over machine data, in our case, logs.
Now lets describe how the different modules of the Onesait Platform can connect to it and how can any other application do so.
Connection to the platform
These are the components that Graylog define as ingestion origins. In our case, we will be using a GELF (Graylog Extended Log Format) TCP input.
The Graylog streams are a mechanism to route messages into categories in real-time while they are processed. You define rules that instruct Graylog which message to route into which streams. In our case, each log has some custom fields from which we can classify their origin. The main custom field we have defined is “app_name” that contains the name of the module writing that log. To make it easier and faster to filter, we have create one Stream for each momdule:
As we have discussed earlier, we can define a series of custom fields in the appenders. This will not only allow us to filter data in streams, but also to be used in our searches. For instance, for the FlowEngine module we have defined the custom field “domain” that will have 3 different values
app.js: Main domain NodeJS process manager. It is the one in charge of managing each NodeRED instance running on the platform.
Proxy-nodered.js: Internal Proxy used by app.js to route to each NodeRED instance
“DomainName”: Lets imagine we have created a FlowEngine domain called “myUserDomain”. If we search by “domain”:”myUserDomain” we will be able to see only our NodeRED instance logs:
Al fields, custom or not, can be used for searching, visualizing or filtering:
Connecting other systems
Once Graylog is up and running, we can use it to add logs from other applications developed from the platform, such as microservices.
Sidecars are a lightweight configuration management system for different log collectors, also called Backends. The Graylog node(s) act as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service (Windows host) or daemon (Linux host).
The log collector configurations are centrally managed through the Graylog web interface. Periodically, the Sidecar daemon will fetch all relevant configurations for the target. These sidecars activate collectors (supporting FileBeat, Winlogbeat y NXLog by default) who will send the logs back to Graylog