From version 3.1.0 in the Control Panel, Ontologies have been renamed as Entities. This does not alter any functionality, simply the nomenclature has been changed for a better understanding of the concept.
The information that comes from the intake and processing is stored in the Semantic Data Hub of the platform, as we have said is the platform's persistence engine.
It supports the storage of information in different stores repositories (currently supported MongoDB, Relational Databases, Elasticsearch, Kudu, BigQuery, HIVE, CosmosDB and at query level any JDBC-driven DB) in addition to always allowing to query in SQL whatever the underlying repository.
To achieve this, the Platform proposes a Data Centric Architecture:
that is, an architecture where data is the main and permanent asset, while applications come and go. In these architectures the data model precedes the implementation of any given application and will be valid long after they are replaced.
On the platform it is supported through the concept of Entity (also called Ontology).
This concept is what allows the platform to be isolated from underlying persistence technologies by offering a layer of SQL abstraction.
The Platform allows, depending on the requirements of each project (volumes of information in real and historical time, mostly read or write access, more analytical processes, previously existing technologies in customers, etc.) select the persistence engine more adequately, conceptually these concepts are handled on platform:
Real Time Database (RealTimeDB): It is designed to support a large volume of inserts and online queries very efficiently. The platform abstracts from the underlying technology allowing to use BD as Mongo, Elastic, Kudu, relational BD, ...
Historical and Analytical Database (HistoricalDB): It is designed to store all the information that is no longer part of the online world (for example, information from previous years that is no longer consulted) and to support analytical processes that extract knowledge of this data (algorithms). The platform supports technologies such as HIVE (with Impala or Presto for online consultations), SAP HANA, BigQuery, CosmosDB, SparkSQL, ...
Staging Area: It allows you to store raw and unprocessed files by the platform, for later intake. HDFS is usually used as storage although the platform allows others like GridFS to be used.
GIS Database: It allows the storage of geographic information and its subsequent consultation and processing, the platform supports MongoDB, Elastic, PostGIS among others.
All the definition and management of these entities is done from the Control Panel of the Platform:
List of Entities / Ontologies (seen by Administrator Role) with visibility levels, ...
Entity Creation Wizard (Ontology), as you can see the platform offers help to create these in a guided way, allowing you to upload a file from which the platform will extract the information to model the entity and upload this file, step by step creation, connection with a relational database:
Creation of the Entity: when I create the entity I can give it a name, tag it, activate or deactivate it, make it public, assign access permissions (who can see it, who can insert data, ...), define audit, ...
If I select Creation step by step, I can define the Entity's attributes and their type (these types are transparently mapped by the platform towards the selected persistence: HIVE, Mongo, Elastic, relational BD, ...)
We can also see how one of the templates can be selected as the basis for the creation of my entity (this is optional) and how you can select an attribute to be encrypted, in this case the platform will store this encrypted attribute in the selected repository and allow decrypt it at the time of returning it.
When I believe the Entity can do an advanced management of the Entity:
In which I can map the entity to a Kafka topic, indicate if the entity's data should be automatically deleted after a while or go to the historical database and indicate the repository on which I wish to store the Entity, in the example they look different repositories, these repositories are configured based on the persistence modules deployed.