HDFS - Cluster

Yarn Hortonworks

About

An HDFS cluster consists of:

  • a single NameNode (the head node) managing the file system. The NameNode is the arbitrator and repository for all HDFS metadata.
  • a number of DataNodes, usually one per node in the cluster performing IO operations on behalves of the namenode.

HDFS has then a head/worker architecture.

A typical deployment has:

  • a dedicated machine that runs only the NameNode software.
  • other machines that runs one instance of the DataNode software.

The system is designed in such a way that user data never flows through the NameNode.

Management

See HDFS - hdfs command line

See the option -printTopology of dfsadmin. It will display a tree of racks and datanodes attached to the tracks as viewed by the NameNode.

Report

See the option -report of dfsadmin.

hdfs dfsadmin -D "fs.default.name=hdfs://10.10.6.20/"  -report
Configured Capacity: 2532916322304 (2.30 TB)
Present Capacity: 2351330590629 (2.14 TB)
DFS Remaining: 2351325233152 (2.14 TB)
DFS Used: 5357477 (5.11 MB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (3):

Name: 10.10.6.14:30010 (wn2-ax.internal.cloudapp.net)
Hostname: wn2-.ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1785765 (1.70 MB)
Non DFS Used: 15721173083 (14.64 GB)
DFS Remaining: 785616138240 (731.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.05%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018


Name: 10.10.6.6:30010 (wn1-ax.internal.cloudapp.net)
Hostname: wn1-ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1789952 (1.71 MB)
Non DFS Used: 18485723136 (17.22 GB)
DFS Remaining: 782851584000 (729.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 92.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018


Name: 10.10.6.5:30010 (wn0-ax.internal.cloudapp.net)
Hostname: wn0-ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1781760 (1.70 MB)
Non DFS Used: 18479804416 (17.21 GB)
DFS Remaining: 782857510912 (729.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 92.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018

Documentation / Reference





Recommended Pages
Yarn Hortonworks
HDFS - DFSAdmin

The DFSAdmin is a sub-command of the hdfs command line and is used for administering an HDFS cluster. These are commands that are used only by an HDFS administrator. dfsadmin is a subcommand of...
Yarn Hortonworks
HDFS - NameNode

NameNode is an HDFS daemon that run on the head node. It' s the head process of the cluster that manages: the file system namespace and regulates access to files by clients. The NameNode: executes...
Yarn Hortonworks
Hadoop Distributed File System (HDFS)

Hadoop Distributed File System (HDFS) Hadoop Distributed File System (HDFS) is a clustered file system. See for an architectural overview. * * * * * * * Amazon...



Share this page:
Follow us:
Task Runner