Spark - Connection (Context)

Card Puncher Data Processing


A Spark Connection is :

This object is called:

An instance of a context object is a spark app

Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program).

A Spark Connection object:

  • connect to a cluster managers (which allocate resources across applications)
  • acquires executors on nodes in the cluster
  • sends your application code (defined by JAR or Python files passed to SparkContext) to the executors.
  • sends tasks to the executors to run.

Every driver (Spark program) has a single Spark Connection object.




  • using the interactive Spark shell, this object is automatically created for you and given the name sc.
  • writing an application, you need to create one yourself.

master connection url

The master connection URL is by default in the configuration file.

Discover More
Card Puncher Data Processing

Map reduce and streaming framework in memory. See: . The library entry point of which is also a connection object is called a session (known also as context). Component: DAG scheduler, ...
Spark Program
Spark - Application

An application is an instance of a driver created via the initialization of a spark context (RDD) or a spark session (Data Set) This instance can be created via: a whole script (called batch mode)...
Spark Cluster
Spark - Cluster

A cluster in Spark has the following component: A spark application composed of a driver program which include the SparkContext (for RDD) or the Spark Session for a data frame which connect to a cluster...
Spark Cluster
Spark - Driver

The driver is a (daemon|service) wrapper created when you get a spark context (connection) that look after the lifecycle of the Spark job. cluster managerapplication manager The driver: start as its...
Card Puncher Data Processing
Spark - Livy (Rest API )

Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing: snippets of code or programs in a Spark Context that runs locally or in YARN. It's used...
Card Puncher Data Processing
Spark - Shell

The shell in different language. A shell create normally automatically a context (connection, session) given the name sc. Example of path: /usr/local/bin/spark-1.3.1-bin-hadoop2.6/bin/ ...
Card Puncher Data Processing
Spark - SparkR API

SparkR provides a distributed data frame implementation that supports operations like selection, filtering, aggregation etc. (similar to R data frames, dplyr) but on large datasets. SparkR also supports...
Card Puncher Data Processing
Spark DataSet - Session (SparkSession|SQLContext)

A org/apache/spark/sql/SparkSessionSpark session is a connection object. It's the application entry point and is used to create a dataset (ie therefore also a dataframe). Previously, it was known as the...
Spark Pipeline
Spark RDD - Spark Context (sc, sparkContext)

A Spark Context is a connection object for an RDD. from a sparksession API pyspark.SparkContext
Card Puncher Data Processing
Sparklyr - Connection (Context)

How to connect to R with Sparklyr. Connect and get a context by giving a master connection URL Livy. Experimental See connections...

Share this page:
Follow us:
Task Runner