Yarn - Application (app)

Yarn Hortonworks

About

An Hadoop application in the context of Yarn is either:

  • a single job (ie a run of an application)
  • or a DAG of jobs.

On a application level (vs cluster level), Yarn consists of:

Yarn is able to run different types of applications.

Type

Type of application that Yarn can run.

Properties

Name

To create an application name:

yarn app -save <Application Name> <File Name>

where:

  • -updateLifetime and -changeQueue alter the values provided in the file.
  • -appTypes specify which client implementation to use

Management

Via the application module of the yarn cli

See the source code entry at ApplicationCLI.java

Kill

Yarn - Yarn CLI

yarn -kill application_id

Stop

yarn -stop <Application Name or ID>

Status

yarn app -status <ApplicationId or ApplicationName>

Launch

yarn app -launch <Application Name> -updateLifetime and -changeQueue -appTypes <Types> <File Name>

# or to start a saved application
yarn start <Application Name>

# or with YARN code bundled in a jar file 
yarn jar <jar> [mainClass] args...

where:

  • -updateLifetime and -changeQueue alter the values provided in the file.
  • -appTypes specify which client implementation to use

If a queue is in STOPPED state, new applications cannot be submitted.

If an app specifically lists a queue in a container resource request, the request is submitted to that queue.

A yarn app is submitted programmatically with a YarnClient interface. See Yarn - App Dev

Move to queue

Yarn - Yarn CLI move to queue

yarn app -changeQueue <Queue Name>
# movetoqueue is Deprecated
#yarn app -movetoqueue <Application ID>

For the fairScheulder, an attempt to move an application to a queue will fail if the addition of the app’s resources to that queue would violate the its maxRunningApps or maxResources constraints.

Modify the number of container

change the number of components/containers running for an application

yarn app -flex <Application Name or ID> -component <Component Name> <Count>

Develop

Yarn - App Dev

List

Yarn - Yarn CLI

yarn app -list

Filter:

  • by type -appTypes <Types>
  • by tag -appTags <Tags>
  • by state -appStates <States>

See also top

Top

yarn top
YARN top - 14:00:34, up 11d, 2:41, 0 active users, queue(s): root
NodeManager(s): 3 total, 3 active, 0 unhealthy, 0 decommissioned, 0 lost, 0 reb
Queue(s) Applications: 20 running, 759 submitted, 0 pending, 739 completed, 0 k
Queue(s) Mem(GB): 153 available, 146 allocated, 0 pending, 0 reserved
Queue(s) VCores: 44 available, 50 allocated, 0 pending, 0 reserved
Queue(s) Containers: 50 allocated, 0 pending, 0 reserved

                  APPLICATIONID USER             TYPE      QUEUE   #CONT  #RCON
 application_1522322322495_0274 sshuser    informatica blaze    default      12
 application_1522230994452_0133 admin       mapreduce    default       5
 application_1522322322495_0687 admin       mapreduce    default       2
 application_1522322322495_0684 admin       mapreduce    default       2
 application_1522322322495_0682 admin       mapreduce    default       2
 application_1522307230623_0033 admin       mapreduce    default       2
 application_1522307230623_0060 admin       mapreduce    default       2
 application_1522307230623_0058 admin       mapreduce    default       2
 application_1522307230623_0020 admin       mapreduce    default       2
 application_1522307230623_0019 admin       mapreduce    default       2
 application_1522307230623_0018 admin       mapreduce    default       2
 application_1522307230623_0016 admin       mapreduce    default       2
 application_1522322322495_0233 admin       mapreduce    default       2
 application_1522322322495_0232 admin       mapreduce    default       2
 application_1522322322495_0231 admin       mapreduce    default       2
 application_1522322322495_0237 admin       mapreduce    default       2
 application_1522322322495_0235 admin       mapreduce    default       2
 application_1522322322495_0726 admin           spark    default       1
 application_1521626910869_0002 hive            spark  thriftsvr       1
 application_1521626910869_0001 hive            spark  thriftsvr       1

Log

yarn logs -applicationId application_1520929888341_0011

History

See:





Discover More
Yarn Hortonworks
Hadoop - Jar (Launch a java application)

How to launch a jar application on hadoop. Please use “yarn jar” to launch YARN applications, not this command.
Tez Hive Query Diagnostic
Hive - Query (SELECT)

in Hive. The SQL language of Hive is called hiveql. ...
Card Puncher Data Processing
Hive - WebHCat (Rest API for HCatalog)

WebHCat ((or Templeton) service is a REST operation based API for HCatalog. WebHCat provides a service that you can use to run Hadoop MapReduce (or YARN), Pig, Hive jobs or perform Hive metadata operations...
Mapreduce Pipeline
MapReduce - Application

Applications: specify the input/output locations supply map and reduce functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise...
Mapreduce Pipeline
MapReduce - Job

A MapReduce Job is a running instance of an MapReduce application This job is a Yarn job as Yarn is the new Hadoop implementation of the Map reduce framework (v2). A map reduce application (/ job) is...
Yarn Hortonworks
Yarn (Yet Another Resource Negotiator) - Hadoop Operating System

Yarn can be seen as the distributed operating system of Hadoop where all apps are build on top of it. Image comes from Hortonworks YARN was originally proposed...
Yarn Hortonworks
Yarn - App Dev

Yarn application development page The yarn application model development has thee main entries: the YarnClient to communicate between the Client to ResourceManager and submit an application....
Yarn Hortonworks
Yarn - Application Master Container (AM) - Job tracker

An Application Master (AM) is a per-application daemon to look after the lifecycle of the job. driver The Application Master daemon is created when an application is started in the very first container....
Yarn Hortonworks
Yarn - Applications Manager (ASM)

The ApplicationsManager is responsible for: accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster restarting the ApplicationMaster...
Yarn Hortonworks
Yarn - Architecture / Topology / Components

The Yarn (ie MapReduce) framework consists of three topology level: federation of independent cluster cluster application (that the Yarn cluster runs) Federation allow multiple independent...



Share this page:
Follow us:
Task Runner