Hive - Live Long And Process (LLAP)

Card Puncher Data Processing

About

Hive LLAP:

  • is slider service (long-lived daemon)
  • supports in-memory data caching
  • was Introduce in Hive 2.0

Functionality such as caching, pre-fetching, some query processing and access control are moved into the daemon. Small/short queries are largely processed by this daemon directly, while any heavy lifting will be performed in standard YARN containers.

Code: llap-server

Management

Configuration

curl localhost:15002/conf 
  • Configurations for LLAP monitoring are stored in resources.json, appConfig.json, metainfo.xml which are embedded into templates.py used by Slider.

Status

  • With web service
curl localhost:15002/status
{
  "status" : "STARTED",
  "uptime" : 139093,
  "build" : "2.1.0-SNAPSHOT from 77474581df4016e3899a986e079513087a945674 by gopal source checksum a9caa5faad5906d5139c33619f1368bb"
}

  • With client
hive --service llapstatus --name {llap_app_name} [-f] [-w] [-i] [-t]

Jmx

  • With web service JSON JMX data
curl localhost:15002/jmx

The LLAP Metrics Collection Server collects JMX metrics from all LLAP Daemons periodically. LLAP Monitor Daemon runs on YARN container, similar to LLAP Daemon, and listens on the same port.

Stacks

  • With web service JVM Stack Traces of all threads -
curl localhost:15002/stacks

Peers (Daemon /)

The list of LLAP Daemons are extracted from the Zookeeper server which launched in the cluster.

  • With web service
curl localhost:15002/peers
{
  "dynamic" : true,
  "identity" : "718264f1-722e-40f1-8265-ac25587bf336",
  "peers" : [
 {
    "identity" : "940d6838-4dd7-4e85-95cc-5a6a2c537c04",
    "host" : "sandbox121.hortonworks.com",
    "management-port" : 15004,
    "rpc-port" : 15001,
    "shuffle-port" : 15551,
    "resource" : {
      "vcores" : 24,
      "memory" : 128000
    },
    "host" : "sandbox121.hortonworks.com"
  },
]
}

Documentation / Reference





Discover More
Azure Cluster
Azure - Cluster (HdInsight Cluster)

Cluster of computer. !!! duplicate of !!! Template reference Each cluster has: an Azure Storage account ...
Card Puncher Data Processing
Hive - Version

See
Card Puncher Data Processing
Spark - Jar

Jar can be defined in a spark-submit command via Jar file with the: --jars option. It define the path to jars file that will be automatically transferred to the cluster. Maven coordinates: --package...



Share this page:
Follow us:
Task Runner