Spark - Client

Card Puncher Data Processing

Spark - Client


Remote Spark installation

On a local node:

  • Download Spark with the same spark and hadoop versions than on the cluster.
  • Extract them to a location (SPARK_HOME)
  • Install the same Java than on the cluster
  • Get the cluster conf file (core-site.xml and yarn-site.xml)
export HADOOP_CONF_DIR=<directory with core-site.xml and yarn-site.xml files>
export SPARK_HOME=<directory where spark archive was extracted>

Share this page:
Follow us:
Task Runner