Map Reduce - Job History Server

Mapreduce Pipeline

About

The history server REST API's allow the user to get status on finished job.

It was known as:

It supports only MapReduce job as seen in the API.

For Spark, see Spark - History Server

Access

Port default to 19888

Rest API

Example: http://historyServerName:port/ws/v1/history/info

<historyInfo>
<startedOn>1541011578573</startedOn>
<hadoopVersion>2.7.0-mapr-1707</hadoopVersion>
<hadoopBuildVersion>
2.7.0-mapr-1707 from f4fe612bcdb0bac9065751ed360a89e4874dd967 by root source checksum c2ec0edf2eb7bb86ec45d03158b3ed
</hadoopBuildVersion>
<hadoopVersionBuiltOn>2017-07-28T16:38Z</hadoopVersionBuiltOn>
</historyInfo>

UI

Jobhistory Ui

Configuration

Env

Map Reduce Job History Server can be configured with the env variable: HADOOP_JOB_HISTORYSERVER_OPTS

Start/Stop

  • Start
${HADOOP_PREFIX}/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver
  • Stop
${HADOOP_PREFIX}/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

Property List

  • yarn.log-aggregation.retain-seconds Default: -1, How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node.
  • yarn.log-aggregation.retain-check-interval-seconds. Default: -1, Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node.

Documentation / Reference





Discover More
Yarn Hortonworks
Hue - Yarn

Yarn in Hue. It is a managed service of Hue. Job Browser Job...
Card Puncher Data Processing
Spark - History Server

job history server history server It lists the following jobs type: incomplete completed attempts. URL web interface: by default For azure hdinsight: ...
Yarn Hortonworks
Yarn - Job

A job is a running instance of an application and can be monitored via: the application master when running the History Server when finished See JobContext...
Yarn Hortonworks
Yarn - Timeline Server (ie yarn app history server)

The storage and retrieval of application’s current and historic information Component: timeline store known as: ApplicationHistoryServer Map Reduce History Server resource manager Rest API...



Share this page:
Follow us:
Task Runner