Yarn - Capacity Scheduler

Yarn Hortonworks

The hierarchy is defined through level in the path yarn.scheduler.capacity.. For instance: The a level (yarn.scheduler.capacity.root.a.queues) is a level below root (yarn.scheduler.capacity.root.queues) * Several queues can be defined by level. * Children do not inherit properties ===== Configuration Management ===== ==== capacity-scheduler.xml ==== The configuration follows the hadoop configuration principles and are stored in the file capacity-scheduler.xml Example with 3 levels (root, a and b) and 8 queues (a,b,c,a1,a2,b1,b2,b3) yarn.scheduler.capacity.root.queuesa,b,cThe queues at the this level (root is the root queue).yarn.scheduler.capacity.root.a.queuesa1,a2The queues at the this level (root is the root queue).yarn.scheduler.capacity.root.b.queuesb1,b2,b3The queues at the this level (root is the root queue). More Configuration ==== Modification ==== === via Cli === Yarn - Yarn CLI vi HADOOP_CONF_DIR/capacity-scheduler.xmlHADOOP_YARN_HOME/bin/yarn rmadmin -refreshQueues === via Rest === Yarn - ResourceManager Rest API https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Scheduler_Configuration_Mutation_API ===== Documentation / Reference ===== * https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_yarn-resource-management/content/enabling_capacity_scheduler.html * https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html




Discover More
Card Puncher Data Processing
Hive - Spark Engine

Hive on Spark provides Hive with the ability to utilize Apache Spark as its execution engine Instead of the capacity scheduler, the fair scheduler is required. ConfigurationProperties...
Yarn Hortonworks
Yarn - Scheduler (S)

The scheduler is a component of the Resource Manager. The Scheduler is responsible for allocating resources to the applications. It performs its scheduling function based on the resource requirements...



Share this page:
Follow us:
Task Runner