Spark - History Server

Card Puncher Data Processing


job history server

For map reduce job, see history server

It lists the following jobs type:

  • incomplete
  • completed
  • attempts.





web interface: http://server:18080 by default

For azure hdinsight:


(see spark.history.provider below),

file-system provider

  • Server side configuration: When using the file-system provider class a base logging directory will contain sub-directories that each represents an application’s event logs.
  • Client Side configuration: The spark jobs themselves must be configured to log events, and to log them to the same shared, writable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:
spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode/shared/spark-logs

Documentation / Reference

Discover More
Jobhistory Ui
Map Reduce - Job History Server

The history server REST API's allow the user to get status on finished job. It was known as: the Yarn History Server the Map Reduce History Server It supports only MapReduce job as seen in the...

Share this page:
Follow us:
Task Runner