Spark - Log

Management

Directory

export SPARK_LOG_DIR=/var/log/spark2

Cluster

Standalone

Spark - Standalone installation (spark scheme) or Spark - Local Installation

Spark executor logs are located in the $SPARK_HOME/work/app-<AppName> directory (where <AppName> is the name of your application.

YARN

See Yarn - Log (Container, Application) - Tfile

Driver logs are by default printed to console


Powered by ComboStrap