Spark - Worker Node

About

A worker node in a cluster (machine)

Worker can also mean executor

Management

Get Hostname

import socket
def getHostname(x):
    return socket.gethostname()
num_worker=3
# A rdd with three partitions
rdd = sc.parallelize(range(1,num_worker+1),num_worker)
rdd.map(getHostname).distinct().sortBy(lambda x: x).collect()
['wn0-hddev2', 'wn1-hddev2', 'wn4-hddev2']

Powered by ComboStrap