Yarn - Localization

About

Localization is the process of copying/download remote resources onto the local file-system of each node.

Instead of always accessing a resource remotely, it is copied to the local machine which can then be accessed locally.

Articles Related

Glossary

• Localizer: The actual thread or process that does Localization. There are two types of Localizers – PublicLocalizer for PUBLIC resources and ContainerLocalizers for PRIVATE and APPLICATION resources.

• LocalCache: NodeManager maintains and manages serveral local-cache of all the files downloaded. The resources are uniquely identified based on the remote-url originally used while copying that file.

LocalResource

LocalResource represents a file/library required to run a container.

The NodeManager is responsible for localizing the resource prior to launching the container.

There is three type of local resource that are localized in the following target directories, under each local-directory:

• PUBLIC: <local-dir>/filecache
• PRIVATE: <local-dir>/usercache//filecache
• APPLICATION: <local-dir>/usercache//appcache/<app-id>/

Documentation / Reference

• https://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/