Introduction

The Platform's FlowEngine component offers a large number of connectors to work with social networks. Specifically, it includes two components: one to "listen" to what happens on Twitter (for example, receiving tweets about a hashtag, or doing a search) and another one to publish tweets:

Open

In this post, you will see how to use these components to:

• Listen to what is being said on Twitter about a topic.

• Store the information of a tweet in Platform.

• Post a tweet with information received from an external service (OpenWeatherMaps).

Step 1: Create APP in Portal Developer on Twitter

The first thing you must do is create an account on Twitter. Once the account is created, go to the Twitter Developer Portal at https://developer.twitter.com/

Open

Once there, go to the Apps section and once there, select Create an App:

It will ask you to confirm that you want to create a developer account, then to complete a set of data. After completing those, you will receive an email to confirm that you want to create a developer account:

After this, you can create your App and get the Keys:

As indicated on the web, you must save these Keys, since you will have to configure them in the FlowEngine components:

After this, you can test the Twitter REST API by pressing the Test an endpoint button.

As indicated, you can test it with the command:

curl -X GET -H "Authorization: Bearer <BEARER TOKEN>" "https://api.twitter.com/2/tweets/440322224407314432"

After testing this, go to the Dashboard. There, you can configure your App.

Go to the Keys and tokens section and select Generate Access Token and Secret:

You must also save this, together with the API Key and the API Key Secret:

IMPORTANT: Until a few months ago, this was enough to finish the necessary configuration in Twitter; but, with the change in the access policy to the Twitter APIs, when registering, you must do it with the Essential level that does not allow access to API 1.1, which is the one that uses the Node-Red component of the FlowEngine.

This means that, for the communication with Twitter to work, you will have to go to the Portal and request the Apply for Elevated which will give you permission to access the API v1.1:

It will ask you for some information to enable access to this level, such as the use you want to make. You can indicate that you want to try to collect tweets with Node-red.

Step 2: Configure Twitter Component in Flowengine and retrieve and post Tweets

The first thing will be to have a user in the Platform instance that you want to use. For the example, you will use the Platform CloudLab environment (free environment for experimentation purposes). So, go to https://lab.onesaitplatform.com/ and if you don't have a user, create it with the SIGN UP button. Bear in mind that you must create a DEVELOPER type user:

With the user created (You will receive an email to validate the creation), you can now log in to the Platform Control Panel:

Once there, go to the DEVELOPMENT > My Flows menu option:

If this is the first time that you enter, you will see that you have not created a FlowEngine domain, so press the + button to create it:

Give it a name and click Create:

Now you can see your domain, which is stopped by default - so, firstly, start it:

Once started, you can access it with the View It button:

Finally, access the FlowEngine Palette. Start by selecting that you do not want to create a project: 

You can maximize the work area with the button in the right corner:

Now, search for the Twitter components with the browser, and drag the Twitter In component to the canvas: 

The triangle means that you must configure it. To do this, click twice on it so that the configuration window appears.

In it, you must configure the Twitter credentials that you had obtained in step 1:

Then, you can configure what you want to listen to. For example, you can choose a trend. In this example, you are going to choose #TheBatman on the day of its premiere in Spain:

Then, configure the debug node to debug the input through the console:

And to test it, simply do a DEPLOY:

And you will already be seeing, in streaming, the tweets on the topic:

To stop it, you can remove the search criteria.

You can also publish a tweet with the Twitter Out component:

In the FlowEngine, it is very easy to build a business logic so that, when certain events happen (the arrival of an event to an Entity/Ontology for example), a message is composed and published.

In this simple example, you will simply post a text. To do that, use the Inject component configured to inject a text:

After this, make the deployment and press the button in the Inject component.

After this, you can see the message:

Step 3: Store Tweet Information in a Platform Entity

If you go back to the example where we was listening to what was being said about The Batman:

and put a debug component configured to output the entire msg object:

then you will see, in the debug tab, a structure like this:

To insert this data in the Platform, you can create an Entity with these attributes: topic, payload, lang, tweet; or you can use the feature of the Platform to create an Entity from a JSON. This second way is much easier when you start from a JSON, as is this case.

From the Debug tab, select one of the objects with the Copy value button:

If you paste the content into an editor, you will see something like this:

Since you want to use this JSON as an example to create our Entity, you can remove the attributes inside the tweet, so you will have a JSON structure like this:

Now go to the DEVELOPMENT/My Entities option in the Control Panel and select the + button to create a new Entity:

There, you will find several options. In this case, you are interested in the Create From File option to use the JSON as a basis to create your Entity:

When selecting it, you can copy the JSON that you already have as a template, then press the Generate JSON-Schema button:

Thenm click Create and complete the data of your Entity:

Once created, you can go back to the FlowEngine to continue with your flow.

Look for the onesaitplatform-insert component and drag it in, then configure it with your Entity:

If you connect it directly to the Twitter output and debug the output, you will see that you get an error:

This is because the output of the components in the FlowEngine is the msg.payload, which in your case is simply a text:

while the Insert node expects a complete JSON structure as you have defined, since you are interested in storing all the tweet info, including the tweeting user.

The way to solve this is to include a Function node between the two, like this:

where the Function node converts the msg to a payload on output:

If you now do the DEPLOY, you will see that the insertion is correct:

To verify that this information is really being stored in an Entity, access the Query Tool in TOOLS>Query Tool.

From there, you can query the stored data, just by selecting the Entity and configuring a query, or by using the default query:

Next steps

Once we have the data stored in the Platform, you can, for example:

• Exploit the data with the Dashboard Engine, for example, showing tweets by zone, users who tweet the most about a topic, …

• Create a Model with the Notebooks doing a sentiment analysis on the tweets, or a geographical distribution, …

• Publish a REST API with the API Manager, with added information on a specific topic.