About
The log of an application (ie from all the containers that the app use when running).
Articles Related
Format
Application logs are not saved in text format. They are saved in a binary format called TFile.
A TFile format is a container of key value pairs, and it is not readable by humans.
Management
View
Command Line
yarn logs -applicationId <applicationId>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application> -containerId <containerId> -nodeAddress <worker-node-address>
where:
Code: LogsCLI.java
Ui
With HortonWorks Ambari > Yarn > Quick Links:
Other tfile reader
tfile reader:
- https://github.com/rkanter/ALFExploder - explode any YARN application aggregated log files into their original separate log files.
Configuration
Aggregation
Logs for all the containers belonging to a single Application are aggregated and written out to a single (possibly compressed) log file at a configured location.
yarn.log-aggregation-enable to true
If disabled, node manager will keep the logs locally
log-dirs
The log should be under the yarn.nodemanager.log-dirs configuration properties
Screenshot from Ambari
/mnt/resource/hadoop/yarn/log
Location
Application
The application log will be at: yarn.nodemanager.log-dirs/application_appid
Container
The Containers log will be at yarn.nodemanager.log-dirs/application_appid/container_{contid}
Framework
| Application Framework | Description |
|---|---|
| Map reduce job | Each container directory will contain the files stderr, stdin, and syslog generated by that container. |
| Other frameworks | They can choose to write more or less files not the location |
Archive
For clusters with a lot of Yarn aggregated logs, it can be helpful to combine them into hadoop archives in order to reduce the number of small files, and hence the stress on the NameNode.
- Usage
mapred archive-logs
More … Hadoop Archive Logs Guide