How to use properties files from Dataflow instances

Each Dataflow instance can configure a property file where developers can add custom properties for their pipelines. This guide explains how to do it.

Persist Streamsets configuration directory

To use this capability, the Streamsets containers managed by the Dataflow must have a persisted volume for their configuration files. These files are in the directory /etc/sdc inside the container.

Create the property file

Create a new file, for example, pipelines.properties in /etc/sdc directory inside the container that you are configuring. Add the properties that you need in this file, for example:

Now you have to link this property file with the Straemsets configuration. Edit the file /etc/sdc/sdc.properties and change the value of the property runtime.conf.location. If you used the name pipelines.properties, then this property must have the next value:

Each time you add a new property to the pipelines.properties file you need to restart the streamsets instance where you add it.

Using properties from Dataflow pipelines

After these steps, you can use the following expression in any form field in pipelines to use a property value:

For example:

Open