Spark - History Server

About

job history server

For map reduce job, see history server

It lists the following jobs type:

  • incomplete
  • completed
  • attempts.

Management

Start

./sbin/start-history-server.sh

URL

Configuration

(see spark.history.provider below),

file-system provider

  • Server side configuration: When using the file-system provider class a base logging directory will contain sub-directories that each represents an application’s event logs.
  • Client Side configuration: The spark jobs themselves must be configured to log events, and to log them to the same shared, writable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:
spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode/shared/spark-logs

Documentation / Reference


Powered by ComboStrap