Spark - Spark-submit

About

The spark submit application to submit application.

The spark-submit script is used to launch applications on a cluster.

Spark jobs are generally submitted from an edge node

Articles Related

Syntax

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --jars <additional1.jar,additional2.jar> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

where:

• class is the main class of the jar
• master is the connection URL. It can be:
  • Spark - Yarn
• deploy-mode can be client or cluster. See Yarn mode
• jar is Spark - Jar
• application-jar is the packaged application.

Example

Run application locally on 8 cores

./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master local[8] \
  /path/to/examples.jar \
  100

More ?. See Java submit

Documentation / Reference

• https://spark.apache.org/docs/latest/submitting-applications.html
• Class: org.apache.spark.deploy.SparkSubmit.main