List
Remote Spark installation
On a local node:
- Download Spark with the same spark and hadoop versions than on the cluster.
- Extract them to a location (SPARK_HOME)
- Install the same Java than on the cluster
- Get the cluster conf file (core-site.xml and yarn-site.xml)
export HADOOP_CONF_DIR=<directory with core-site.xml and yarn-site.xml files>
export SPARK_HOME=<directory where spark archive was extracted>
- Spark - Spark-submit should work.