Spark - Connection (Context)

About

A Spark Connection is :

• a context object (known also as connection)
• the first step when creating a script

This object is called:

• an SQL Context for a RDD (in Spark 1.x.)
• SparkSession for a dataset (dataframe). (It was also known as the SqlContext)

An instance of a context object is a spark app

Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program).

A Spark Connection object:

• connect to a cluster managers (which allocate resources across applications)
• acquires executors on nodes in the cluster
• sends your application code (defined by JAR or Python files passed to SparkContext) to the executors.
• sends tasks to the executors to run.

Every driver (Spark program) has a single Spark Connection object.

Articles Related

Management

Initialization

When:

• using the interactive Spark shell, this object is automatically created for you and given the name sc.
• writing an application, you need to create one yourself.

master connection url

The master connection URL is by default in the configuration file.