Introduction

The new Process Verifier asset allows to check the result of the execution of processes developed on the platform allowing to verify the correct loading of data into the platform.

In addition to this, the Process Verifier allows us to add rules to these processes for a better control of the inserted data, for example a rule that checks the correct integrity of the loaded data (dataExist) or one that allows us to control the number of loaded records (volumetry).

Process Verifier Concepts

In the diagram we can see the flow that the Process Verifier follows:

Open

Logbook Entity

This entity contains the information about the processes to be verified.

Open

This Entity stores the information about the processes to be launched as well as the rules to be applied, its attributes are:

• process: the process name will be recorded in this field.

• date: refers to the process field that stores the date of data insertion.

• granularity: frequency of inserted data.

• temporality: frequency of data insertion.

• day_week: day of the week on which the data loading is performed (Monday = 1, Sunday = 7).

• day_month: day of the month on which the data is loaded.

• monthly_mismatch: difference of months between the data load and the information it references.

• weekly_mismatch: difference of weeks between the data load and the information it is referenced.

• daily_mismatch: daily difference between the data load and the information it references.

• comments: comments that we can add on the process for more information.

• rules: field containing the definition of the rules that apply to each process. This field is further divided as follows:

  • type: type of rule that applies to this process.

  • records: expected records for this process (only applies in "volumetric" rule).

  • tolerance: expected records tolerance (only applies in "volumetric" rule).

Registry Entity

This entity stores the execution record of the processes and whether they have passed the different rules defined for them:

• process: process name.

• date: date when the verification was performed.

• type: type of rule that has been verified.

• detail: information in case of verification error.

• status: record of the process status for a certain rule ("Passed" or "Failed").

How to use the Process Verifier?

Main Flow

To use the Process Verifier I will create a flow in the FlowEngine like this:

In the Main flow we will have a manual trigger that will allow us to launch the process on request. The image shows how to execute it on a specific date:

In addition, the process can be dynamically triggered by the "trigger dynamic" node once a day.

In the parameterization nodes we define the following variables:

• date: today's date.

• ont: log entity from which we will obtain the information.

• ontLog: log entity where we will insert the information.

Rule execution flow

In this flow all the necessary configuration is defined before verifying the different rules of each process:

The "query parameterization" node is used to create the query necessary to obtain the information from each process and store it in the variable "msg.payload".

The "payload to JSON" node simply transforms the query output to a JSON, while the "split payload" node separates this JSON into different objects.

After this, we must perform the separation of rules, this is done in the "split rules" node, where we create a new variable ( msg.rule ) that we will use in the next "switch" node to separate the objects and apply the rule that applies to it.

After going through the flow of the rule to be applied, we will insert the result in the register entity.

Rules:

After separating by rules we enter the definition of each of them in their respective flows. We can add as many rules as we wish, initially these two are included:

dataExist Rule

In this rule we check the existence of data for the granularity specified in the process, if there is no data, the rule will be given as "Failed" and a message explaining the reason for the error will be reflected in the "detail" field.

Otherwise the rule will remain as "Passed".

Volumetry rule

This rule is used to check the volume of data recorded for a specific process. The expected number of records and the tolerance (percentage number) is given within the same rule.

In case of receiving a number of records that is within the expected number, the rule will be given as "Passed", but otherwise it will remain as "Failed" and a message explaining the failure will be added in the "detail" field.

Example of execution

Next, we will show an example of the execution of this verifier for these processes:

Example code for Flow Engine

In this ZIP you will find the code of the asset to import into the Platform.

Open

To import it we will simply download the file and we will go to the "Bundle Load" section inside "Version Control" in Platform.

Once in this page we will click on the "Upload ZIP" button and then on "Import".

Definition Process 1:

• process: Process1

• date: datetime

• fromat: yyyy/mm/dd

• granularity: daily

• temporality: weekly

• day_week: 0

• day_month: 0

• monthly_mismatch: 0

• weekly_mismatch: 0

• daily_mismatch: 0

• rules:

  • Item 1

    • type: dataExist

  • Item 2

    • type: volumetry

    • records: 336

    • tolerance: 20

Definición Process 2:

• process: Process2

• date: datetime

• fromat: yyyy/mm/dd

• granularity: daily

• temporality: weekly

• day_week: 0

• day_month: 0

• monthly_mismatch: 0

• weekly_mismatch: 0

• daily_mismatch: 0

• rules:

  • Item 1

    • type: dataExist

  • Item 2

    • type: volumetry

    • records: 1

    • tolerance: 20

For this case there will be no data from process 1 on 12/14/2023, besides that the total number of records inserted from process 2 is 66.

After launching our process checker the result in the record entity will be the following: