Azure - HDInsight (Microsoft's Hadoop)

Card Puncher Data Processing


Azure HDInsight is a cluster distribution of the Hadoop components from the Hortonworks Data Platform (HDP).

duplicate of Azure - Cluster (HdInsight Cluster)

It regroups open-source frameworks:

  • Hadoop,
  • Spark,
  • Hive,
  • LLAP,
  • Kafka,
  • Storm,
  • R,
  • and more.


Post-install script

Install component at the end of the creation with script action

Example: install hue

Add an edge node. See Edge Node


  • On a headnode (mycluster is the name of the cluster)
# List the root
hdfs dfs -D "" -ls /
# Report
hdfs dfsadmin -D "" -report
  • Check the integrity of HDFS on the HDInsight cluster by using the following commands:
hdfs fsck -D "" /
Connecting to namenode via
FSCK started by hdsshadm (auth:SIMPLE) from / for path / at Thu Jan 17 16:04:30 UTC 2019
................................Status: HEALTHY
 Total size:    1200 B
 Total dirs:    139
 Total files:   32
 Total symlinks:                0 (Files currently being written: 15)
 Total blocks (validated):      15 (avg. block size 80 B)
 Minimally replicated blocks:   15 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          5
 Number of racks:               1
FSCK ended at Thu Jan 17 16:04:30 UTC 2019 in 12 milliseconds

The filesystem under path '/' is HEALTHY

hdfs dfsadmin -D "" -safemode leave


From SSH Connection

  • To headnode
ssh -i ~/.ssh/myPrivatekey -p 22 [email protected] # Primary HeadNode
ssh -i ~/.ssh/myPrivatekey -p 23 [email protected] # Secondary HeadNode
  • To edge
ssh [email protected]
  • To WorkerNode (from head or edge node). If the SSH account is secured using SSH keys, make sure that ssh forwarding is enabled on the client.
ssh sshuser@wn0-myhdi

Discover More
Yarn Hortonworks
(Apache) Hadoop

Apache Hadoop is basically 3 things: a distributed file system (HDFS), a distributed operating system (Yarn) and map reduce applications Image comes from Hortonworks...
Azure Cluster
Azure - Cluster (HdInsight Cluster)

Cluster of computer. !!! duplicate of !!! Template reference Each cluster has: an Azure Storage account ...
Card Puncher Data Processing
Azure - Enterprise Security Package (HDInsight Premium)

Enterprise Security Package (previously known as HDInsight Premium) Apache Ranger (Enterprise Security Package) Domain joining (Enterprise Security Package) Secure shell (SSH) access ...
Azure Storage Structure
Azure - Windows Azure Storage Blob (WASB) - HDFS

Windows Azure Storage Blob (WASB) is an file system implemented as an extension built on top of the HDFS APIs and is in many ways HDFS. The WASB variation uses: SSL certificates for improved security...

Share this page:
Follow us:
Task Runner