HDFS - URI

About

What is a URI (Uniform resource identifier)? in HDFS to locate a file.

Articles Related

Syntax

The URI format is [scheme://][authority]/path where:

the scheme is optional. See scheme
the path can be relative to the working directory

Example:

hadoop fs -cat hdfs://namenodehost1/file1 hdfs://namenodehost2/file2

where:

HDFS - NameNode

Management

Scheme

The default is specified in the core-site.xml file.

List:

hdfs for HDFS
file for the local file system: file:/// (default)
HFTP FS,
S3 FS
wasb - Azure Blob Storage
adls - Azure Data Lake Storage

Modify

Core-site.xml

If you want to modify it, add the following to Hadoop - core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Get with hdfs

hdfs getconf -confKey fs.defaultFS

wasb://[email protected]

Hadoop Cli

Every configuration can be change at the command line with the D option.

Example:

hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls /

Java

Configuration conf = new Configuration();
String hdfsUri = "hdfs://mycluster/";
conf.set("fs.defaultFS", hdfsUri);
FileSystem fileSystem = FileSystem.get(URI.create(hdfsUri), conf);

with the following import

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;