Spark on your own machine : Différence entre versions

Version actuelle en date du 4 mars 2016 à 14:07

Installing and setting

Java

 $ sudo apt-add-repository ppa:webupd8team/java
 $ sudo apt-get update
 $ sudo apt-get install oracle-java7-installer

Scala

 $ wget http://www.scala-lang.org/files/archive/scala-2.10.4.tgz
 $ sudo mkdir /usr/local/src/scala
 $ sudo tar xvf scala-2.10.4.tgz -C /usr/local/src/scala/

Spark

 $ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.2-bin-hadoop2.6.tgz
 $ sudo mkdir /usr/lib/spark
 $ sudo tar xvf spark-1.5.2-bin-hadoop2.6.tgz -C /usr/lib/spark

SBT

 $ wget https://dl.bintray.com/sbt/native-packages/sbt/0.13.9/sbt-0.13.9.tgz 
 $ sudo tar xvf sbt-0.13.9.tgz -C /usr/lib/

Add the following lines in the .bashrc file:

 export JAVA_HOME=/usr/lib/jvm/java-7-oracle
 export SCALA_HOME=/usr/local/src/scala/scala-2.10.4
 export SPARK_HOME=/usr/lib/spark/spark-1.5.2-bin-hadoop2.6
 export SBT_HOME=/usr/lib/sbt
 export PATH=$SCALA_HOME/bin:$JAVA_HOME/bin:$SPARK_HOME/bin:$SBT_HOME/bin:$PATH

$ source ~/.bashrc (to reload the file .bashrc)

Scala IDE for Eclipse

http://scala-ide.org/download/sdk.html

Create a Spark-SBT project

- Write your application

 $ mkdir -p project1/src/main/scala 
 $ cd project1/
 $ echo " " > src/main/scala/SimpleApp.scala
 $ gedit src/main/scala/SimpleApp.scala &

- Create the build definition of the program (build.sbt) and add the following lines:

 name := "Simple Project"
 
 version := "1.0"
 
 scalaVersion := "2.10.4"
 
 libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.2"

- Create the JAR:

 $ sbt package

- Run the project via spark-submit :

 $ spark-submit --class "SimpleApp" --master local[2] target/scala-2.10/simple-project_2.10-1.0.jar

Use an Editor

- Add sbteclipse to your plugin definition file (or create one if doesn't exist). You can use either:

 - the global file (for version 0.13 and up) at ~/.sbt/0.13/plugins/plugins.sbt
 - the project-specific file at PROJECT_DIR/project/plugins.sbt

For the latest version:

 addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")

- Run the command eclipse

 $ sbt eclipse

- Import your project on Eclipse : import > General > Existing project into workspace > your-project