HDFS - Block

About

File System - Block in HDFS.

The block size can be changed by file.

Block are stored on a datanode and are grouped in block pool

Management

Info

Location

The location on where the blocks are stored is defined in hdfs-site.xml. Example:

<property>
	<name>dfs.datanode.data.dir</name>
	<value>file:/hadoop/data/dfs/datanode</value>
</property>

Offset

Size

A typical block size used by HDFS is 128 MB. Thus, an HDFS file is chopped up into 128 MB chunks.

<property>
  <name>dfs.blocksize</name>
  <value>134217728</value>
</property>
hdfs getconf -confKey dfs.blocksize
134217728
# of 128 Mb

Move

See the mover hdfs sub-command to move block replicas across storage types.

hdfs mover

Failure

Under-replicated

under-replicated block

  • web ui: The overview page gives you this information.

_

Missing


Powered by ComboStrap