Map Reduce - Job History Server
About
The history server REST API's allow the user to get status on finished job.
It was known as:
- the Yarn History Server To get history info about Yarn app, see Yarn - Timeline Server (ie yarn app history server)
- the Map Reduce History Server
It supports only MapReduce job as seen in the API.
For Spark, see Spark - History Server
Articles Related
Access
Port default to 19888
Rest API
Example: http://historyServerName:port/ws/v1/history/info
<historyInfo>
<startedOn>1541011578573</startedOn>
<hadoopVersion>2.7.0-mapr-1707</hadoopVersion>
<hadoopBuildVersion>
2.7.0-mapr-1707 from f4fe612bcdb0bac9065751ed360a89e4874dd967 by root source checksum c2ec0edf2eb7bb86ec45d03158b3ed
</hadoopBuildVersion>
<hadoopVersionBuiltOn>2017-07-28T16:38Z</hadoopVersionBuiltOn>
</historyInfo>
UI
Configuration
Env
Map Reduce Job History Server can be configured with the env variable: HADOOP_JOB_HISTORYSERVER_OPTS
Start/Stop
- Start
${HADOOP_PREFIX}/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver
- Stop
${HADOOP_PREFIX}/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver
Property List
- yarn.log-aggregation.retain-seconds Default: -1, How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node.
- yarn.log-aggregation.retain-check-interval-seconds. Default: -1, Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node.