Spark - Web UI (Driver UI)

Card Puncher Data Processing

About

Each driver program has a web UI, typically on port 4040, that displays information about:

The Spark UI will tell you which DataFrames and what percentages are in memory. Spark - (Executor) Cache

This UI is only available for the duration of the application. The cluster manager UI's (master en worker) are always available. Example with the built-in standalone cluster manager, the UI by default are available at the port 8080 for the master and 8081 for the workers.

Management

URL

http://<driver-node>:4040

where:




Discover More
Card Puncher Data Processing
Spark

Map reduce and streaming framework in memory. See: . The library entry point of which is also a connection object is called a session (known also as context). Component: DAG scheduler, ...
Spark Caching
Spark - (Executor) Cache

in Spark. Each executor has a cache. from the lines is recomputed lines is NOT recomputed but get from the cache
Card Puncher Data Processing
Spark - Configuration

The configuration of Spark is mostly: configuration around an app. runtime-environment The application web UI...
Spark Cluster
Spark - Driver

The driver is a (daemon|service) wrapper created when you get a spark context (connection) that look after the lifecycle of the Spark job. cluster managerapplication manager The driver: start as its...
Card Puncher Data Processing
Spark - Monitoring

The web UI, typically on port 4040, displays information : running tasks, executors, and storage usage.



Share this page:
Follow us:
Task Runner