Spark - Client

List

Remote Spark installation

On a local node:

  • Download Spark with the same spark and hadoop versions than on the cluster.
  • Extract them to a location ($SPARK_HOME)
  • Install the same Java than on the cluster
  • Get the cluster conf file (core-site.xml and yarn-site.xml)
export HADOOP_CONF_DIR=<directory with core-site.xml and yarn-site.xml files>
export SPARK_HOME=<directory where spark archive was extracted>

Powered by ComboStrap