HDFS - Configuration (hdfs-site.xml)

Yarn Hortonworks


HDFS follows the same configuration scheme than the whole Hadoop platform. See Hadoop - Configuration (HADOOP_CONF_DIR)

The configuration are split between two files:

  • hdfs-site.xml, which provides default behaviors for the HDFS client.
  • core-site.xml, which sets the default filesystem name.

The hdfs configurations are stored in:


Get a value

With the hdfs utility, to get the value of a key:

hdfs getconf -confKey key

Example with the file system URI. You get the value here of a wasb azure file system

hdfs getconf -confKey fs.defaultFS
wasb://[email protected]


hdfs envvars | grep -i HADOOP_CONF_DIR

See HDFS - Environment Variable

Pass a value at runtime

  • Command line: You can pass a configuration at the command line with the D generic option.
hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls /
  • Code API (Java)
Configuration conf = new Configuration();
String hdfsUri = "hdfs://mycluster/";
conf.set("fs.defaultFS", hdfsUri);
FileSystem fileSystem = FileSystem.get(URI.create(hdfsUri), conf);

Discover More
Yarn Hortonworks
HDFS - (User) Authentication, Identification

in HDFS. See also: adminusers User identity mechanism is specified by the configuration property: hadoop.security.authentication simple: same as OS kerberos The user is the Linux user....
Yarn Hortonworks

POSIX style permissions/HDFS ACLs in HDFS is one authorization method . By default, ACLs are disabled. dfs.namenode.acls.enabled - Set to true to enable support for HDFS ACLs (Access Control...
Hdfs Ui Block Information
HDFS - Block

in HDFS. The block size can be changed by file. Block are stored on a datanode and are grouped in block pool The location on where the blocks are stored is defined in hdfs-site.xml....
Yarn Hortonworks
HDFS - Block Replication

in HDFS HDFS stores each file as a sequence of blocks. The blocks of a file are replicated for fault tolerance. The NameNode makes all decisions regarding replication of blocks. It periodically receives...
Yarn Hortonworks
HDFS - Checkpoint

During a checkpoint the changes from the transaction log (Editlog) are applied to the metadata store (FsImage) (because it's not efficient to record each change on the metadata store (FsImage) When...
Hadoop Hdfs Fsimage
HDFS - FsImage File

The HDFS file system metadata are stored in a file called the FsImage. It contains: the entire file system namespace the mapping of blocks to files and file system properties The FsImage...
Yarn Hortonworks
HDFS - Heartbeat (Dead datanode)

Each DataNode sends a Heartbeat message to the NameNode periodically. The NameNode marks DataNodes without recent Heartbeats as dead and does not forward any new IO requests to them. The time-out...
Yarn Hortonworks
HDFS - NameNode

NameNode is an HDFS daemon that run on the head node. It' s the head process of the cluster that manages: the file system namespace and regulates access to files by clients. The NameNode: executes...
Yarn Hortonworks
HDFS - Port

Example for the private port of an Azure cluster Service Nodes Port Protocol Description NameNode web UI Head nodes 30070 HTTPS Web UI to view status NameNode metadata service head nodes 8020...
Hdfs Namenode Ui
HDFS - Web UI (Namenode UI)

A typical HDFS install configures a client web server to navigate the HDFS namespace and view the contents of its files. Azure: Service Nodes Port Protocol Description NameNode web UI Head...

Share this page:
Follow us:
Task Runner