{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"style":{"value":"none"}},"macroMetadata":{"macroId":{"value":"fc7d33e2-0884-41ae-8cda-b4ee6b2f7047"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"04479b83-b542-4a99-99aa-c8a9f16c5761"}},{"type":"heading","attrs":{"level":1},"content":[{"text":"Introduction","type":"text"}]},{"type":"paragraph","content":[{"text":"SparkR","type":"text","marks":[{"type":"strong"}]},{"text":" is an R package that provides a lightweight front end for using Apache Spark from within R. SparkR provides an implementation of distributed data frameworks that supports operations such as selection, filtering, aggregation, etc. (similar to R's data frameworks, dplyr) but on large datasets. SparkR also supports distributed machine learning via MLlib.","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://spark.apache.org/docs/latest/sparkr.html"}},{"type":"paragraph","content":[{"text":"Onesait Platform Notebooks include the ","type":"text"},{"text":"SparkR interpreter","type":"text","marks":[{"type":"strong"}]},{"text":" by default, so it is not necessary to install anything beforehand, just include the ","type":"text"},{"text":"%spark.r","type":"text","marks":[{"type":"code"}]},{"text":" header.","type":"text"}]},{"type":"paragraph","content":[{"text":"Below are some examples of how to use SparkR to calculate and represent results.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Examples of use","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Hello World","type":"text"}]},{"type":"paragraph","content":[{"text":"The first example consists of displaying a few variables of different types in the results window. To do this, the following code will be written in a paragraph:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nfoo <- TRUE\nprint(foo)\nbare <- c(1, 2.5, 4)\nprint(bare)\ndouble <- 15.0\nprint(double)","type":"text"}]},{"type":"paragraph","content":[{"text":"When the paragraph is executed, the result will be displayed.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":350,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":358,"alt":"image-20250228-092259.png","id":"5a735865-503d-49c5-b455-9103961b5927","collection":"contentId-4982702092","type":"file","height":277}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Loading R Libraries","type":"text"}]},{"type":"paragraph","content":[{"text":"It is possible to load R elements as libraries, which allows you to extend the functionality of the interpreter. For example, it is possible to load the table library to display data in tabular form:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nlibrary(data.table)\ndt <- data.table(Cabecera = 1:3)\nprint(dt)\nfor (i in 1:5) {\n print(i*2)\n}\nprint(1:50)","type":"text"}]},{"type":"paragraph","content":[{"text":"In this case, the following will be displayed when executing the paragraph:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":600,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":562,"alt":"image-20250228-093031.png","id":"a673ffa4-d0f0-44bf-abe2-a31cf84ab2f6","collection":"contentId-4982702092","type":"file","height":425}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Loading and interacting with Datasets","type":"text"}]},{"type":"paragraph","content":[{"text":"It is possible to load datasets in order to interact with them. We will take for example the ‘iris’ dataset, a dataset that comes integrated with R, and that contains information about iris flower species.","type":"text"}]},{"type":"paragraph","content":[{"text":"To display the data header of the dataset, the following code will be executed:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\ncolnames(iris)\niris$Petal.Length\niris$Sepal.Length","type":"text"}]},{"type":"paragraph","content":[{"text":"After executing the above paragraph, the indicated information of the set will be displayed:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":550,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":532,"alt":"image-20250228-093245.png","id":"3c94d498-a897-4fbf-9a27-8bf33f6b1684","collection":"contentId-4982702092","type":"file","height":324}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Creating a Spark DataFrame","type":"text"}]},{"type":"paragraph","content":[{"text":"As with other libraries, it is possible to create a DataFrame in which to perform complex operations. In the following example we are going to read information from a CSV, process the original data and then display it.","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark\n\nimport org.apache.commons.io.IOUtils\nimport java.net.URL\nimport java.nio.charset.Charset\n\nval bankText = sc.parallelize(\n IOUtils.toString(\n new URL(\"https://raw.githubusercontent.com/apache/zeppelin/master/testing/resources/bank.csv\"),\n Charset.forName(\"utf8\")).split(\"\\n\"))\n\ncase class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)\n\nval bank = bankText.map(s => s.split(\";\")).filter(s => s(0) != \"\\\"age\\\"\").map(\n s => Bank(s(0).toInt, \n s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\", \"\"),\n s(3).replaceAll(\"\\\"\", \"\"),\n s(5).replaceAll(\"\\\"\", \"\").toInt\n )\n).toDF()\nbank.registerTempTable(\"bank\")","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"In this case ","type":"text"},{"text":"%spark","type":"text","marks":[{"type":"code"}]},{"text":" and not ","type":"text"},{"text":"%spark.r","type":"text","marks":[{"type":"code"}]},{"text":" is used as the interpreter.","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Read Spark DataFrame from R","type":"text"}]},{"type":"paragraph","content":[{"text":"In this example we are going to read the previously created DataFrame. To do so, this paragraph will be generated:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\ndf <- sql(\"select count(*) from bank\")\nprintSchema(df)\nSparkR::head(df)","type":"text"}]},{"type":"paragraph","content":[{"text":"The result of executing the paragraph will be as follows:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":600,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":574,"alt":"image-20250228-101854.png","id":"042ace38-0c68-410c-8f21-af6c66314cbf","collection":"contentId-4982702092","type":"file","height":257}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Creating an R DataFrame","type":"text"}]},{"type":"paragraph","content":[{"text":"It is also possible to create an R DataFrame. This will be done as follows:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r \n\nlocalNames <- data.frame(name=c(\"John\", \"Smith\", \"Sarah\"), budget=c(19, 53, 18))\nnames <- createDataFrame(localNames)\nprintSchema(names)\nregisterTempTable(names, \"names\")","type":"text"}]},{"type":"paragraph","content":[{"text":"The result of executing this paragraph will show the schema of the DataFrame:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":500,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":553,"alt":"image-20250228-102615.png","id":"b33e6fd1-c700-4d95-bcb8-09c02b432cc3","collection":"contentId-4982702092","type":"file","height":273}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Read R DataFrame from Spark (Scala)","type":"text"}]},{"type":"paragraph","content":[{"text":"It is possible to access and read this DataFrame from Spark. To do so, this paragraph will be created:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark\n\nsqlContext.sql(\"select * from names\").head","type":"text"}]},{"type":"paragraph","content":[{"text":"After executing it, the following will be displayed:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":400,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":366,"alt":"image-20250228-102817.png","id":"12cc0ff2-b69d-4df2-8f57-bc973e4ed9ed","collection":"contentId-4982702092","type":"file","height":141}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Querying the R DataFrame with SQL","type":"text"}]},{"type":"paragraph","content":[{"text":"It is also possible to analyse the DataFrame using SQL, which will allow you to visualise the results:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.sql\n\nselect * from names","type":"text"}]},{"type":"paragraph","content":[{"text":"The result will look like this:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":700,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":1164,"alt":"image-20250228-103222.png","id":"e1bbc9d1-25f6-4663-9f29-c6584a203048","collection":"contentId-4982702092","type":"file","height":437}}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Depending on the display method last chosen, it will be displayed in one form or another: table, bar chart, pie chart, etc. This can be selected in the representation type bar:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":250,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":225,"alt":"image-20250228-103408.png","id":"6ed03979-b52b-4b73-a98f-e398a8c9cb63","collection":"contentId-4982702092","type":"file","height":80}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Pair chart","type":"text"}]},{"type":"paragraph","content":[{"text":"To analyse the distribution of data, a pair chart can be drawn in a simple way. Thus, taking the data of the ‘iris’ set as a starting point:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\npairs(iris)","type":"text"}]},{"type":"paragraph","content":[{"text":"The result would be:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":700,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":801,"alt":"image-20250228-103633.png","id":"e7c81ca8-1c8b-469a-9afb-67f7838a8caf","collection":"contentId-4982702092","type":"file","height":946}}]},{"type":"paragraph","content":[{"text":"It is possible to represent this graph with colour ranges by slightly modifying the code. Thus, for a range of three colours it would be specified as follows:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nplot(iris, col = heat.colors(3))","type":"text"}]},{"type":"paragraph","content":[{"text":"The result would look like this:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":700,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":803,"alt":"image-20250228-103802.png","id":"edf48322-b838-45a6-8c62-f86c06c84e45","collection":"contentId-4982702092","type":"file","height":948}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Heat map","type":"text"}]},{"type":"paragraph","content":[{"text":"Another typical graphic that can be represented is the heat map. It can be generated using the following code:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nlibrary(ggplot2)\npres_rating <- data.frame(\n rating = as.numeric(presidents),\n year = as.numeric(floor(time(presidents))),\n quarter = as.numeric(cycle(presidents))\n)\np <- ggplot(pres_rating, aes(x=year, y=quarter, fill=rating))\np + geom_raster()","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":700,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":799,"alt":"image-20250228-104540.png","id":"656e5bbf-b7c7-4cc4-bfbe-07b2577e9a23","collection":"contentId-4982702092","type":"file","height":1059}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Visualisation of Graphs","type":"text"}]},{"type":"paragraph","content":[{"text":"There are also different ways to represent data in graphical form. One of them is using the ","type":"text"},{"text":"Google Charts API","type":"text","marks":[{"type":"strong"}]},{"text":" (googleVis).","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://cran.r-project.org/web/packages/googleVis/vignettes/googleVis_intro.html"}},{"type":"paragraph","content":[{"text":"Examples include:","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Bar Chart","type":"text"}]},{"type":"paragraph","content":[{"text":"Defining a DataFrame with the values of coordinates and abscissae, as shown in the following code:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nlibrary(googleVis)\ndf=data.frame(country=c(\"US\", \"GB\", \"BR\"), \n val1=c(10,13,14), \n val2=c(23,12,32))\nBar <- gvisBarChart(df)\nprint(Bar, tag = 'chart')","type":"text"}]},{"type":"paragraph","content":[{"text":"The resulting graph will look like the following:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":550,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":515,"alt":"image-20250228-104811.png","id":"c49d2546-4e70-4492-98d5-a1276e28e5de","collection":"contentId-4982702092","type":"file","height":426}}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Candle Diagrams","type":"text"}]},{"type":"paragraph","content":[{"text":"This type of diagram can be generated with the following code:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nlibrary(googleVis)\n\nCandle <- gvisCandlestickChart(OpenClose, \n options=list(legend='none'))\n\nprint(Candle, tag = 'chart')","type":"text"}]},{"type":"paragraph","content":[{"text":"The result of executing the paragraph would be as follows:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":500,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":448,"alt":"image-20250228-104938.png","id":"e007e5e0-8479-4ab0-b296-736a935cc214","collection":"contentId-4982702092","type":"file","height":409}}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Line Chart","type":"text"}]},{"type":"paragraph","content":[{"text":"In a very similar way to the bar chart, a line chart can be generated with the following code:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nlibrary(googleVis)\ndf=data.frame(country=c(\"US\", \"GB\", \"BR\"), \n val1=c(10,13,14), \n val2=c(23,12,32))\n\nLine <- gvisLineChart(df)\n\nprint(Line, tag = 'chart')","type":"text"}]},{"type":"paragraph","content":[{"text":"After running it, it will look like this:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":500,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":509,"alt":"image-20250228-105056.png","id":"1fdbae5f-e7cb-404f-8fba-c8aedd08b24f","collection":"contentId-4982702092","type":"file","height":452}}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Bubble Diagram","type":"text"}]},{"type":"paragraph","content":[{"text":"The information can also be visualised in the form of a bubble diagram. Taking the ‘fruits’ dataset, the diagram would be generated using the following code:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nlibrary(googleVis)\nbubble <- gvisBubbleChart(Fruits, idvar=\"Fruit\", \n xvar=\"Sales\", yvar=\"Expenses\",\n colorvar=\"Year\", sizevar=\"Profit\",\n options=list(\n hAxis='{minValue:75, maxValue:125}'))\nprint(bubble, tag = 'chart')","type":"text"}]},{"type":"paragraph","content":[{"text":"The result would be the following graph:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":500,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":470,"alt":"image-20250228-105214.png","id":"64654ba0-85a7-4ce6-9a5a-73d89fba5ce2","collection":"contentId-4982702092","type":"file","height":424}}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Maps","type":"text"}]},{"type":"paragraph","content":[{"text":"Last but not least, it is possible to display projected maps. To do so, the following code must be defined:","type":"text"}]},{"type":"codeBlock","content":[{"text":"%spark.r\n\nlibrary(googleVis)\ngeo = gvisGeoChart(Exports, locationvar = \"Country\", colorvar=\"Profit\", options=list(Projection = \"kavrayskiy-vii\"))\nprint(geo, tag = 'chart')","type":"text"}]},{"type":"paragraph","content":[{"text":"In the example shown, a map will then be displayed with the countries coloured according to the ‘exports’ dataset:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":700,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":797,"alt":"image-20250228-105332.png","id":"5c748cdb-e4fc-400a-86ee-4d2b05103e66","collection":"contentId-4982702092","type":"file","height":528}}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Conclusions","type":"text"}]},{"type":"paragraph","content":[{"text":"As can be seen, it is possible to interact with data in different ways using SparkR Notebooks. In the case of more advanced representations, using graphs and diagrams, the different APIs, such as the Google one used in this case, are useful and very practical.","type":"text"}]},{"type":"paragraph","content":[{"text":"This tutorial has shown basic representations as an example, so it is recommended to read the documentation of both SparkR and the APIs that you want to use in order to configure the available options in more detail.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Download example","type":"text"}]},{"type":"paragraph","content":[{"text":"The example used in this tutorial is available for download below:","type":"text"}]},{"type":"mediaGroup","content":[{"type":"media","attrs":{"id":"da2fb3d9-c223-4311-9949-0f70276664e9","collection":"contentId-4982702092","type":"file"}}]},{"type":"paragraph"},{"type":"paragraph"},{"type":"paragraph"},{"type":"paragraph"},{"type":"paragraph"},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"a","type":"text"}]}],"version":1}

Browser not supported