Spark - Worker Node

Spark Cluster


A worker node in a cluster (machine)

Worker can also mean executor

Spark Cluster


Get Hostname

import socket
def getHostname(x):
    return socket.gethostname()
# A rdd with three partitions
rdd = sc.parallelize(range(1,num_worker+1),num_worker) x: x).collect()
['wn0-hddev2', 'wn1-hddev2', 'wn4-hddev2']

Discover More
Spark Cluster
Spark - Cluster

A cluster in Spark has the following component: A spark application composed of a driver program which include the SparkContext (for RDD) or the Spark Session for a data frame which connect to a cluster...

Share this page:
Follow us:
Task Runner