HDFS - DataNode

Yarn Hortonworks

HDFS - DataNode

About

A dataNode is a HDFS process that manage storage attached to the nodes that they run on.

The DataNodes are responsible for serving read and write requests from the file system’s clients. The DataNodes also perform block creation, deletion, and replication upon instruction from the NameNode.

The files are on the dataNode not on the NameNode

Management

List

with HDFS - DFSAdmin

hdfs dfsadmin -D "fs.default.name=hdfs://10.10.6.20/"  -report
Configured Capacity: 2532916322304 (2.30 TB)
Present Capacity: 2351330590629 (2.14 TB)
DFS Remaining: 2351325233152 (2.14 TB)
DFS Used: 5357477 (5.11 MB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (3):

Name: 10.10.6.14:30010 (wn2-ax.internal.cloudapp.net)
Hostname: wn2-.ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1785765 (1.70 MB)
Non DFS Used: 15721173083 (14.64 GB)
DFS Remaining: 785616138240 (731.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.05%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018


Name: 10.10.6.6:30010 (wn1-ax.internal.cloudapp.net)
Hostname: wn1-ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1789952 (1.71 MB)
Non DFS Used: 18485723136 (17.22 GB)
DFS Remaining: 782851584000 (729.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 92.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018


Name: 10.10.6.5:30010 (wn0-ax.internal.cloudapp.net)
Hostname: wn0-ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1781760 (1.70 MB)
Non DFS Used: 18479804416 (17.21 GB)
DFS Remaining: 782857510912 (729.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 92.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018

State

See HDFS DataNode Admin Guide

With HDFS - DFSAdmin see the options -getDatanodeInfo

: Get the information about the given datanode. This command can be used for checking if a datanode is alive. ==== Shutdown ==== See the options -shutdownDatanode [upgrade]'' of HDFS - DFSAdmin





Discover More
Hdfs Ui Block Information
HDFS - Block

in HDFS. The block size can be changed by file. Block are stored on a datanode and are grouped in block pool The location on where the blocks are stored is defined in hdfs-site.xml....
Hdfs Ui Block Pool Id
HDFS - Block Pool

A block pool is a collection of block (on a datanode ???) See the options -deleteBlockPool of dfsadmin. where: Snpashot from the Overview of Snapshot from ...
Yarn Hortonworks
HDFS - Block Replication

in HDFS HDFS stores each file as a sequence of blocks. The blocks of a file are replicated for fault tolerance. The NameNode makes all decisions regarding replication of blocks. It periodically receives...
Yarn Hortonworks
HDFS - Blockreport

A blockreport is a list of all HDFS data blocks that correspond to each of the local files, and sends this report to the NameNode. Each datanode create and send this report to the namenode: when the...
Yarn Hortonworks
HDFS - Cluster

An HDFS cluster consists of: a single NameNode (the head node) managing the file system. The NameNode is the arbitrator and repository for all HDFS metadata. a number of DataNodes, usually one per...
Yarn Hortonworks
HDFS - DFSAdmin

The DFSAdmin is a sub-command of the hdfs command line and is used for administering an HDFS cluster. These are commands that are used only by an HDFS administrator. dfsadmin is a subcommand of...
Hdfs Datanode Ui
HDFS - Datanode Web UI

datanode Web ui. where you can see: Service Nodes Port Protocol Description DataNode All worker nodes 30075 HTTPS Web UI to view status, logs, etc. URL Each data node...
Yarn Hortonworks
HDFS - Heartbeat (Dead datanode)

Each DataNode sends a Heartbeat message to the NameNode periodically. The NameNode marks DataNodes without recent Heartbeats as dead and does not forward any new IO requests to them. The time-out...
Yarn Hortonworks
HDFS - NameNode

NameNode is an HDFS daemon that run on the head node. It' s the head process of the cluster that manages: the file system namespace and regulates access to files by clients. The NameNode: executes...
Yarn Hortonworks
HDFS - Port

Example for the private port of an Azure cluster Service Nodes Port Protocol Description NameNode web UI Head nodes 30070 HTTPS Web UI to view status NameNode metadata service head nodes 8020...



Share this page:
Follow us:
Task Runner