Introduction

An Entity in the RealTime database (RealTimeDB) can store a large amount of information depending on the purpose of the Entity, such as an Entity that receives events from a network of hundreds of devices.

To avoid problems of memory collapse with queries that do not apply limit criteria on Entities with a large number of records, the maximum number of records that such queries will return can be configured at the installation level.

Thus, for a generic query such as:

A maximum of 2,000 records will be returned, regardless of whether the Entity has millions of records.

To facilitate the processing of this type of sentences in projects, an operation is included in Digital Broker to allow the execution of sentences without limit clauses or with very high limits, the result of which will be dumped to a file that will be made available to users.

How is it used?

The operation available in the Digital Broker is located in the path /rest/ontology/{ontology}/file, which has the following interface:

Open

The parameters to be filled in are the following:

• ontology: Entity on which the query is made, and in which the user must have permissions.

• query: query to be executed.

• queryType: type of query to be performed (SQL or native).

• responseType: destination of the file to be generated:

  • DISK: the file will be created in an installation-configurable directory within the Semantic Inf Broker container, which can be mapped to a shared volume so that the file is available for other modules, such as DataFlow, Notebooks, etc.

  • S3_MINIO: the file is generated in the MinIO bucket of the Digital Client owner.

  • URL: the file is created in a directory mapped to a shared folder in the Platform loadbalancer, allowing it to be downloaded via a URL.

The response from the service will be a JSON with the information where to locate the file:

• Type Disk:

Open

• Type URL:

• Type S3 (MinIO):

The file remains accessible in the user's bucket:

Asynchronous Response

The response of the service is asynchronous, since in queries on large Entities the query time can be long. To this end, the response JSON includes an identifier that allows the execution status of the sentence to be consulted, which can be in:

• IN_PROGRESS: the sentence is being executed.

• FINISHED: the instruction has finished and its result has been completely dumped to the file.

The operation that allows consulting the status is available in the Digital Broker in the path /rest/ontology/file/{queryId}/status, and has the following interface:

An example of how it works is as follows:

Files Deletion

In case the result is not stored in S3 (MinIO), the file will remain accessible to the user for the next hour. After that time the file will be deleted and if it has not been previously downloaded, it will be necessary to re-launch the query.