The log of an application (ie from all the containers that the app use when running).
Application logs are not saved in text format. They are saved in a binary format called TFile.
A TFile format is a container of key value pairs, and it is not readable by humans.
yarn logs -applicationId <applicationId>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application> -containerId <containerId> -nodeAddress <worker-node-address>
where:
Code: LogsCLI.java
With HortonWorks Ambari > Yarn > Quick Links:
tfile reader:
Logs for all the containers belonging to a single Application are aggregated and written out to a single (possibly compressed) log file at a configured location.
yarn.log-aggregation-enable to true
If disabled, node manager will keep the logs locally
The log should be under the yarn.nodemanager.log-dirs configuration properties
Screenshot from Ambari
/mnt/resource/hadoop/yarn/log
The application log will be at: yarn.nodemanager.log-dirs/application_appid
The Containers log will be at yarn.nodemanager.log-dirs/application_appid/container_{contid}
Application Framework | Description |
---|---|
Map reduce job | Each container directory will contain the files stderr, stdin, and syslog generated by that container. |
Other frameworks | They can choose to write more or less files not the location |
For clusters with a lot of Yarn aggregated logs, it can be helpful to combine them into hadoop archives in order to reduce the number of small files, and hence the stress on the NameNode.
mapred archive-logs
More … Hadoop Archive Logs Guide