Table of Contents

About

What is a URI (Uniform resource identifier)? in HDFS to locate a file.

Syntax

The URI format is [scheme://][authority]/path where:

Example:

hadoop fs -cat hdfs://namenodehost1/file1 hdfs://namenodehost2/file2

where:

Management

Scheme

The default is specified in the core-site.xml file.

List:

  • hdfs for HDFS
  • file for the local file system: file:/// (default)
  • HFTP FS,
  • S3 FS
  • wasb - Azure Blob Storage
  • adls - Azure Data Lake Storage

Modify

Core-site.xml

If you want to modify it, add the following to Hadoop - core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Get with hdfs

hdfs getconf -confKey fs.defaultFS
wasb://[email protected]

Hadoop Cli

Every configuration can be change at the command line with the D option.

Example:

hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls /

Java

Configuration conf = new Configuration();
String hdfsUri = "hdfs://mycluster/";
conf.set("fs.defaultFS", hdfsUri);
FileSystem fileSystem = FileSystem.get(URI.create(hdfsUri), conf);

with the following import

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;