Yarn - Scheduler (S)

1 - About

The scheduler is a component of the Resource Manager.

The Scheduler is responsible for allocating resources to the applications.

It performs its scheduling function based on the resource requirements of the applications defined in resource Container (which incorporates elements such as memory, cpu, disk, network etc.)

The scheduler:

  • performs no monitoring or tracking of status for the application.
  • offers no guarantees about restarting failed tasks either due to application failure or hardware failures.

3 - Management

3.1 - List

3.1.1 - Default

The default scheduler is the JobQueueTaskScheduler. See Yarn - Default Scheduler

3.1.2 - Plugin

The Scheduler use plugin to implement schedulers functionality. They are responsible for partitioning the cluster resources among the various queues, applications etc.

Built-in Plugin:

3.2 - UI

The ResourceManager web UI/scheduler web-page should show the resource usages of individual queues.

3.3 - Port

  • HTTP: 8030 conf: yarn.resourcemanager.scheduler.address - The address of the scheduler interface
  • HTTP: 8033 conf: yarn.resourcemanager.admin.address - The address of the RM admin interface.

3.4 - Load simulator

Yarn - Scheduler Load Simulator (SLS)

4 - Documentation / Reference


Data Science
Data Analysis
Statistics
Data Science
Linear Algebra Mathematics
Trigonometry

Powered by ComboStrap