Yarn - Application Master Container (AM) - Job tracker

1 - About

An Application Master (AM) is a per-application daemon to look after the lifecycle of the job.

For instance, in Spark, it's called the driver

The Application Master daemon is created when an application is started in the very first container.

The AM:

  • communicates with YARN cluster
  • handles application execution
  • negotiate resources from the ResourceManager
  • work with the NodeManager(s) to execute and monitor the tasks.

It has the responsibility to provide a web UI and to send that link to the RM.

During application launch time, the main tasks of the ApplicationMaster are:

  • a) communicating with the ResourceManager to negotiate and allocate resources for future containers
  • b) after container allocation, communicating YARN NodeManager's (NMs) to launch application containers on them.

3 - Management

3.1 - Rest Api

3.2 - Memory

  • Required AM memory is (asked memory+384 MB)
  • Limit: max threshold yarn.scheduler.maximum-allocation-mb

Data Science
Data Analysis
Data Science
Linear Algebra Mathematics

Powered by ComboStrap