Spark DataSet - Partition

Card Puncher Data Processing

Apply function
  • foreachPartition(func) - Runs func on each partition of this Dataset.
  • mapPartitions - Returns a new Dataset that contains the result of applying func to each partition. Experimental
  • sortWithinPartitions - Returns a new Dataset with each partition sorted by the given expressions. This is the same operation as “SORT BY” in SQL (Hive QL).
  • coalesce - reduce the number of partitions - if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions.
// coalesce the data to a single file.
Repartition creates partitions based on the user’s input by performing a full shuffle on the data (after read) If the number of partitions is not specified, the number is taken from spark.sql.shuffle.partitions.Returns a new Dataset partitioned by the given partitioning expressions. Same as DISTRIBUTE BY in SQL. All rows with the same Distribute By columns will go to the same reducer. However, Distribute By does not guarantee clustering or sorting properties on the distributed keys.
Number of dynamic partitions created is XXXX, which is more than 1000.
Number of dynamic partitions created is 2100, which is more than 1000. 
To solve this try to set hive.exec.max.dynamic.partitions to at least 2100.;

The below configuration must be set before starting the spark application

A set with Spark SQL Server will not work. You need to set the configuration at the start of the server.from
Documentation / Reference

Discover More
Sql Hive Arch
Spark - Hive

Hive is the default Spark catalog. Since Spark 2.0, Spark SQL supports builtin Hive features such as: HiveQL Hive SerDes UDFs read...
Spark Cluster Tasks Slot
Spark - Task

A task is a just thread executed by an executor on a slot (known as core in Spark). The total number of slot is the number of thread available. See . The number of Partitions dictate the number of tasks...
Card Puncher Data Processing
Spark DataSet - Bucket

A partition may be divided in bucket. Buckets the output by the given columns. If specified, the output is laid out on the file system similar to Hive's bucketing scheme. This is applicable...
Spark Query Plan Generation
Spark Engine - Shuffle

shuffle means moving data rows by rows between partition. spark.sql.shuffle.partitions - Configures the number of partitions to use when shuffling data for joins or aggregations.

Share this page:
Follow us:
Task Runner