Spark - Standalone installation (spark scheme)

Standalone is a simple cluster manager included with Spark that makes it easy to set up a cluster.

The connection URL is:

• spark://hostnameMaster:port to connect to a remote standalone spark.

Example with sparklyr:

sc <- sparklyr::spark_connect(master = "spark://nicoLaptop:7077")

where: master = Spark - Master (Connection URL )

• place a compiled version of Spark on each node on the cluster.
• Start a master server:

./sbin/start-master.sh
# The start master script will call the following command
# ./bin/spark-class.cmd org.apache.spark.deploy.master.Master --host HI-LAPTOP-NGD1 --port 7077 --webui-port 8082
# Then
# C:\Program Files\Java\jdk1.8.0_45\jre\bin\java -cp C:/spark-2.1.1-bin-hadoop2.7/conf\;C:\spark-2.1.1-bin-hadoop2.7\bin\..\jars\* -Xmx1g org.apache.spark.deploy.master.Master --host HI-LAPTOP-NGD1 --port 7077 --webui-port 8080

• Start a worker node

./sbin/start-slave.sh <master-spark-URL>

For the worker and the master

The port of the UI by default is:

• 8080 for the master
• 8081 for the worker

It can be change through:

• the start script option –webui-port PORT
• or the env SPARK_MASTER_WEBUI_PORT and SPARK_WORKER_WEBUI_PORT

The port of the service by default is:

• 7077 for master
• random for worker

It can be change through:

• the start script option –port PORT
• or the env SPARK_WORKER_PORT and SPARK_MASTER_PORT

• https://spark.apache.org/docs/latest/spark-standalone.html