Introduction

A new functionality has been added to the Platform that will allow to trace and monitor the execution of a process in the Platform involving several components.

In this first version of the functionality, only operations with Entities (inserts, updates, deletes) from different sources (IoTBroker, Dataflow, etc.) have been taken into account.

How to create a process

To create a new process, access the menu Processing > Process Traceability Definition.

The list of traceability processes available to the user will then be displayed. To create a new one, click on the ‘+’ button at the top right of the screen.

The traceability creation wizard will then be launched, in which the following information has to be filled in:

Identification: the unique identifier of the process.
Description: a short text describing what the process does.
Cron: expression that will define how often the process has to be executed. Clicking on the clock icon opens a dialog box where the cron can be easily programmed.

Activate From/To: it is possible to select a date interval for the execution of the process, during which the monitoring of this process will be launched. If these fields are left empty, it will be executed as long as the Active process field is enabled.
Active Process: indicates whether the traceability of the process is active or not. The status of the process will only be checked if it is active.
Filter by Field: indicates that you want to filter the operations according to a predefined field. This is useful when you have several devices performing operations on the same Entity and you only want to take into account one of them for monitoring the process.
Ordered Execution: indicates whether the operations defined later must be executed in an ordered manner, or whether the order of execution does not matter.
Check Executions: indicates whether or not the number of executions of each operation must be taken into account when verifying the execution of the process. In the following example, it has been configured so that the insert operation has two executions, which means that for this operation to be verified, two instances must be inserted in the indicated Entity.
Add Operation: you can add as many operations as you want to the process. The following properties must be indicated for each operation
- The Type of operation: insert, update, delete.
- The Entity on which the operation will be executed.
- The Field by which you want to filter the operations, which will correspond to the fields of the selected Entity. This option is only enabled when the Filter by Field option is selected, and will not be taken into account in delete type operations. It is important that the selected field is a unique identifier.
- The Default Value that the previously selected field must take, in order for the operation to be validated. Note that the identifier must be an alphanumeric field.
- The Source of the execution: ApiManager, DataFlow, Notebook, etc.
- The Number of Executions, if the option to verify executions is enabled.
- The Position, if the option of ordered execution is enabled.

Once the information on the form has been filled in, all you have to do is click on the ‘Create’ button to finish creating the process. Next, the list of available processes will appear, where the newly created process can be viewed.

How to visualise the traceability of a process

User interface in the Control Panel

Once the process has been created, you can access its visualisation through the list. To do this, click on the ‘👁’ icon in the process options.

A box diagram will appear on one side representing our process:

In the case of an ordered process, the operations will appear united and in order.
In the case of an unordered process, the operations will appear unlinked to each other.

In this diagram, the latest status of each of the operations can be seen visually with a colour code (‘green’ if everything has gone well, and ‘red’ if an error has occurred).

Furthermore, by selecting one of the operations directly on the diagram, it will be possible to obtain information about the operation, as well as its last status and the error message in the event that the execution of the operation has failed.

On the other hand, there is a list with the history of all the executions of the process, broken down by operations and ordered by default in descending order. In the event that one of the operations has failed, an information icon will appear next to the operation and the execution error message will appear.

Audit

Process traceability data is stored in the user's Audit Entity. Therefore, all executions of a process can be queried through the Query Tool, and can even be included in a Dashboard in a visual way.

The data follows the following structure in JSON format:

{
    "extraData": "{\"e01c999e-c998-4db6-b592-865be5ad9713\":{\"message\":\"Operation not executed.\",\"status\":false},\"aa258a0f-08a1-4b24-93e4-a5c7dabadd86\":{\"message\":\"Operation not executed.\",\"status\":false},\"765898a8-59e9-4571-9f7e-c40c8dfda71e\":{\"message\":\"Operation OK\",\"status\":true}}",
    "module": "CONTROLPANEL",
    "message": "Process with identification process04 executed successfuly.",
    "type": "PROCESS_EXECUTION",
    "version": 1,
    "otherType": "com.minsait.onesait.platform.controlpanel.helper.processtrace.ProcessTraceHelper",
    "timeStamp": 1636981555525,
    "formatedTimeStamp": "2021-11-15T13:05:55.525Z",
    "operationType": "PROCESS_EXECUTION",
    "id": "e751faf3-844b-42f6-895e-6af89a7df5df",
    "resultOperation": "SUCCESS",
    "user": "rbarrio1",
    "ontology": "Audit_rbarrio1"
}

As data of interest, we have:

All data with type=PROCESS_EXECUTION refer to the traceability of processes.
The version field indicates the version of the process to which the data corresponds.
In the resultOperation field it is possible to verify if the execution of the process has been successful or not.
The id field corresponds to the process identifier.
The extraData field contains, in JSON format, the information of each operation separately.

Data Versioning

As seen in the previous section, when visualising the traceability data of a process, it is crucial to know the operations involved in the process. For this reason, modifying the operations of a process that has already been active and has data stored in the audit could cause compatibility problems between the data.

To avoid this, a simple data versioning system has been created, so that each time the operations of a process are modified, the version is upgraded, which will be specified in the version field of the audit data.

It should be noted that in the graphical interface of the Control Panel, when the history of the executions of a process is consulted, only the data of the latest version will appear in the list.

How the verification of processes is managed

Non-ordered processes

Once a non-ordered process has been created, periodically (according to the periodicity indicated when the process was created) it will be checked whether the process has been executed correctly or not.

In this case, the operations shall be checked individually, since the operations are not related to each other in any way.

Ordered processes

In this case, when checking the status of the process, start by checking the first operation and if it has been executed correctly, move on to the second operation, and so on. In the event that one of the intermediate operations has failed, the whole process is considered to have failed even if there are subsequent operations that have been executed correctly, since it is understood that the process is only valid if all the operations are executed in the correct order.