Spark - Driver

1 - About

The driver is a (daemon|service) wrapper created when you get a spark context (connection) that look after the lifecycle of the Spark job.

Spark supports many cluster manager and in the context of Yarn, the driver is a synonym for the application manager.

The driver:

  • start as its own service (daemon)
  • connect to a cluster manager,
  • manage them. It listen for and accept incoming connections from its worker (executors) throughout its lifetime. As such, the driver program must be network addressable from the worker nodes.

You can then create your spark application interactively.

Every driver has a single Sparkcontext object.

See Spark - Cluster

3 - Management

3.1 - Memory

Spark - Configuration

  • spark.driver.memory

The driver in Yarn is the application manager. See application manager memory

  • perrmgen is controlled by spark.driver.extraJavaOptions

spark-shell --conf spark.driver.extraJavaOptions="-XX:MaxPermSize=384m"

3.2 - Core

Number of thread (ie core)

Spark - Configuration. spark.driver.cores

3.3 - PID

export SPARK_PID_DIR=/var/run/spark2

3.4 - Service port

Spark - Configuration

Conf Default Desc (local hostname) Hostname or IP address for the driver. This is used for communicating with the executors and the standalone Master.
spark.driver.port (random) Port for the driver to listen on. This is used for communicating with the executors and the standalone Master.

3.5 - UI

Default to 4040

See Spark - Web UI (Driver UI)

3.6 - Machine

The driver machine is the single machine where the driver will run (and therefore initiates the Spark job and where summary results will be collected)

It can be:

  • client: the local machine
  • cluster: the resource manager instantiate a machine

For instance on the Yarn cluster manager, see Yarn deployment mode

Data Science
Data Analysis
Data Science
Linear Algebra Mathematics

Powered by ComboStrap