Spark Engine - Partition

RDD - Partition

in RDD Articles Related Managememnt set parrallelize. (Example for two) get PySpark mapPartitions Return a "...

Spark - Executor (formerly Worker)

When running on a cluster, each Spark application gets an independent set of executor JVMs that only run tasks and store data for that application. Worker or Executor are processes that run computati "...

Spark - Resilient Distributed Datasets (RDDs)

Resilient distributed datasets are one of the data structure in Spark. Write programs in terms of operations on distributed datasets Partitioned collections of objects spread across a cluster, store "...

Spark DataSet - Partition

in Spark may be subdivides in bucket follow the same SQL rule than Hive Partitions The num of Partitions dictate the number of tasks that are launched. The computation is taking place on one node "...