Yarn - Log (Container, Application) - Tfile

About

The log of an application (ie from all the containers that the app use when running).

Format

Application logs are not saved in text format. They are saved in a binary format called TFile.

A TFile format is a container of key value pairs, and it is not readable by humans.

Management

View

Command Line

doc

yarn logs -applicationId <applicationId>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application> -containerId <containerId> -nodeAddress <worker-node-address>

where:

Code: LogsCLI.java

Ui

With HortonWorks Ambari > Yarn > Quick Links:

_

Other tfile reader

tfile reader:

Configuration

Aggregation

Logs for all the containers belonging to a single Application are aggregated and written out to a single (possibly compressed) log file at a configured location.

yarn.log-aggregation-enable to true

If disabled, node manager will keep the logs locally

log-dirs

The log should be under the yarn.nodemanager.log-dirs configuration properties

Screenshot from Ambari

/mnt/resource/hadoop/yarn/log

_

Location

Application

The application log will be at: ${yarn.nodemanager.log-dirs}/application_${appid}

Container

The Containers log will be at ${yarn.nodemanager.log-dirs}/application_${appid}/container_{$contid}

Framework

Application Framework Description
Map reduce job Each container directory will contain the files stderr, stdin, and syslog generated by that container.
Other frameworks They can choose to write more or less files not the location

Archive

For clusters with a lot of Yarn aggregated logs, it can be helpful to combine them into hadoop archives in order to reduce the number of small files, and hence the stress on the NameNode.

  • Usage
mapred archive-logs

More … Hadoop Archive Logs Guide


Powered by ComboStrap