Spark - History Server


job history server

For map reduce job, see history server

It lists the following jobs type:

  • incomplete
  • completed
  • attempts.






(see spark.history.provider below),

file-system provider

  • Server side configuration: When using the file-system provider class a base logging directory will contain sub-directories that each represents an application’s event logs.
  • Client Side configuration: The spark jobs themselves must be configured to log events, and to log them to the same shared, writable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:
spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode/shared/spark-logs

Documentation / Reference

Powered by ComboStrap