ArtsAutosBooksBusinessEducationEntertainmentFamilyFashionFoodGamesGenderHealthHolidaysHomeHubPagesPersonal FinancePetsPoliticsReligionSportsTechnologyTravel
  • »
  • Technology»
  • Computers & Software»
  • Operating Systems»
  • Windows

Installation Spark Cluster on Windows

Updated on March 30, 2016
./sbin/start-master.sh

But on Windows it's more complex. How to do it right, without mistake and don't spend too much time.
Do it step by step and carefully.

Installation

1. Open Command Prompt ( Win+R : cmd )

2. Go to directory of Spark project

cd c:\spark-1.6.0\bin


3. Start master server by executing:

./bin/spark-class org.apache.spark.deploy.master.Master


4. Take HOST/POST of URL: Master will give [spark://HOST:POST - URL] - Address for starting application

5. Initiate the operation of one or more processes of applications and connect them to the master

./bin/spark-class org.apache.spark.deploy.worker.Worker spark://IP:PORT


6. Connect application vs cluster

./bin/spark-shell --master spark:://IP:PORT

Run simple application and view informations

See from Spark Guide: http://spark.apache.org/docs/latest/quick-start.html
Write this code into cmd and run:

scala> val textFile = sc.textFile("c:\\spark-1.6.0\\README.md") //directory of file
scala> textFile.count() // Number of items in this RDD
scala> textFile.first() // First item in this RDD
scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))
scala> textFile.filter(line => line.contains("Spark")).count() // number of lines contain in "Spark"

Video of installation & test application

We received image results as below


images of Spark Master, Spark Job, Spark Environment and Spark Executors.

Spark Master

Spark Jobs

Spark Environment

Spark Executors

Conclusions

This topic isn't new topic for someone but many of us didn't run Spark on Windows. So you don't worry and practice on Windows carefully, do it step by step. Remember commands and way to connect spark app with a cluster. From Web Master UI we get whole information of processes, memory, environment, executors, etc.

References

1. Quick start - Spark-1.6.0: http://spark.apache.org/docs/latest/quick-start.html
2. Spark Standalone Mode: http://spark.apache.org/docs/latest/spark-standalone.html#starting-a-cluster-manually

Comments

    0 of 8192 characters used
    Post Comment

    No comments yet.