Spark

About

Map reduce and streaming framework in memory.

See:

• Spark Engine - Data Structure (DataSet, DataFrame and RDD)
• Spark - Application.

The library entry point of Spark which is also a connection object is called a session (known also as context).

Component:

• DAG scheduler,
• a query optimizer,
• and a physical execution engine.

Spark essentials

• Spark - Driver
  • Spark - (RDD) Transformation,
  • Spark - Action,
  • persistence,
  • Spark - Broadcast variables,
  • Spark - Accumulator,
• Spark - Key-Value RDD,

Framework

The Spark Framework consists of :

• the core Apache Spark component
• Spark SQL,
• Spark Streaming,
• the ML machine learning library (mlib),
• the GraphX graphical computing library.
• Spark Web UI

Language API

• Scala
• Java
• R
• Python (pySpark)

Documentation / Reference

• Analyze accelerometer data with Apache Spark and MLlib
• Spark Papers - Cluster Computing with Working Sets
• Learning Spark: Lightning-Fast Big Data Analysis