HDFS - Block

1 - About

File System - Block in HDFS.

The block size can be changed by file.

Block are stored on a datanode and are grouped in block pool

3 - Management

3.1 - Info

3.2 - Location

The location on where the blocks are stored is defined in hdfs-site.xml. Example:


<property>
	<name>dfs.datanode.data.dir</name>
	<value>file:/hadoop/data/dfs/datanode</value>
</property>

3.2.1 - Offset

3.3 - Size

A typical block size used by HDFS is 128 MB. Thus, an HDFS file is chopped up into 128 MB chunks.


<property>
  <name>dfs.blocksize</name>
  <value>134217728</value>
</property>

  • From the command line:

hdfs getconf -confKey dfs.blocksize


134217728
# of 128 Mb

3.4 - Move

See the mover hdfs sub-command to move block replicas across storage types.


hdfs mover

3.5 - Failure

3.5.1 - Under-replicated

under-replicated block

  • web ui: The overview page gives you this information.

3.5.2 - Missing


Data Science
Data Analysis
Statistics
Data Science
Linear Algebra Mathematics
Trigonometry

Powered by ComboStrap