Using Data Across Interpreters

In this section, we’ll describe how to fetch data in some interpreter and using it in other interpreters with the zeppelin context util.

We’re going to fetch data in onesaitplatform interpreter and using it in %python, %pyspark. %r, %spark

First at all, we’ll connect to the platform with some IoTClient:

Then, we’re going to fetch data from some query a putting it in zeppelin context. This put the results in an array of strings with all the results of the query:

%onesaitplatform
z.put("dataz", select * from Restaurants limit 1)

We can also use a native query (for mongodb ontologies):

%onesaitplatform
z.put("dataznative", db.Restaurants.find())

After that, we’ll can use it in python or pyspark using z.get('key') (sometimes it necessary to use z.z.get('key') or for example in python is necessary to get the first result with [0] to print something).

With this result, we can create some spark dataframe and its temp table from this ontology in spark (scala + spark):

%spark
import scala.util.parsing.json._
var dataRaw = z.get("dataz")
var dataArrayStr = dataRaw.asInstanceOf[Array[String]] //casting to Array[String]
val OPRDD = sc.parallelize(dataArrayStr)
val OPDF = sqlContext.read.json(OPRDD)
OPDF.registerTempTable("OPSparkTempTable")
z.show(OPDF.select("Restaurant.*"))

Or in pyspark (python + spark):

Even we can use SparkSQL interpreter to query this data:

If we want to use this data in pandas dataframe, we can read it in this way (with %python or %pyspark):

We can work in the other way, we can generate data in python and use it in onesaitplatform interpreter to insert data. In python:

And in onesaitplatform interpreter for insert data:

Other way to work is to use onesaitplatform python libs from pypi (onesaitplatform-client-services ) that come with onesaitplatform notebook instalation, Rest APIs of onesaitplatform or the Java API for spark interpreter.