New Configuration Options when using Elasticsearch as the Platform's RealTimeDB

Available from version 3.0.0

The possibilities for configuring ontologies defined on Elasticsearch have been increased:

Shards and Replicas Configuration:

We have added the possibility of not only configuring the default values for these fields, but also being able to set values for each ontology:

Template Configuration:

To store audit data, logs, timeseries, etc. Elasticsearch recommends creating several indexes per time block, especially if the data is more or less homogeneous per time block. For example, create an index per month, or per week, or per day, etc.

To do this, Elasticsearch provides a number of elements that we have also added to the configuration of the ontologies:

  • Elasticsearch allows you to create templates instead of indexes associated to a pattern. For example, if you create a template associated to log-*, then any insertion made in an index that meets the pattern, e.g. log-2021-02-18, will create the index if it does not previously exist, using the mapping configured in the template.

  • Elasticsearch allows you to create one alias or more to make queries, so that an alias indicates several indexes. When querying, the alias is used and the query is executed on all the shards of all the indexes, which is actually exactly the same, performance-wise, as having an index with the same number of total shards. In the templates, you can indicate that, when an index is created, it is included in a specific alias.

In this new version we have added the possibility of creating ontologies as an Elasticsearch template, where you must indicate the criteria for index generation:

As you can see in the previous image, you will have to specify a field used for index generation, as well as a function to apply on it.

The allowed functions are:

  • NONE: Creates the index as the concatenation of the ontology’s name + the field’s value.

  • SUBSTRING: Creates the index as the concatenation of the ontology’s name + substring(field’s value, start, end).

  • Rest of time functions: Extracts the day, month and/or year data based on the selection. These functions require the field to be of type Timestamp.

Say that you have an ontology “Ontology” defined as in the previous field. When inserting a data with the field Ontology.timestamp=”2021-04-08T10:10:00Z”, it will automatically insert the data in the index “Ontology-2021-04”, having as many indexes as years/months in the data. All of this is transparent to the user.

As for the query, when the ontology is created, along with the creation of the template, an alias is assigned with the name of the ontology, and it encompasses all the indexes that can be created according to the previous rule.