HDFS - Configuration (hdfs-site.xml)

1 - About

HDFS follows the same configuration scheme than the whole Hadoop platform. See Hadoop - Configuration (HADOOP_CONF_DIR)

The configuration are split between two files:

  • hdfs-site.xml, which provides default behaviors for the HDFS client.
  • core-site.xml, which sets the default filesystem name.

The hdfs configurations are stored in:

3 - Management

3.1 - Get a value

With the hdfs utility, to get the value of a key:


hdfs getconf -confKey key

Example with the file system URI. You get the value here of a wasb azure file system


hdfs getconf -confKey fs.defaultFS


wasb://[email protected]

3.2 - Directory


hdfs envvars | grep -i HADOOP_CONF_DIR


HADOOP_CONF_DIR='/usr/hdp/2.6.2.25-1/hadoop/conf'

See HDFS - Environment Variable

3.3 - Pass a value at runtime

  • Command line: You can pass a configuration at the command line with the D generic option.

hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls /

  • Code API (Java)

Configuration conf = new Configuration();
String hdfsUri = "hdfs://mycluster/";
conf.set("fs.defaultFS", hdfsUri);
FileSystem fileSystem = FileSystem.get(URI.create(hdfsUri), conf);


Data Science
Data Analysis
Statistics
Data Science
Linear Algebra Mathematics
Trigonometry

Powered by ComboStrap