Spark SQL - Conf (Set)

About

Backlinks of Spark in SQL is made through the SET statement. Same usage as Hive - Set

Articles Related

Example

List

set;

One by one

set hive.exec.dynamic.partition.mode=nonstrict;
set spark.executor.cores=1; -- the number of executor
set spark.dynamicAllocation.enabled=false;
set spark.executor.instances=1; --	The number of executors for static allocation. With spark.dynamicAllocation.enabled, the initial set of executors will be at least this large.
set spark.cores.max=1; -- the maximum amount of CPU cores to request for the application from across the cluster (not from each machine). If not set, the default will be spark.deploy.defaultCores 
-- you control the degree of parallelism post-shuffle using �SET spark.sql.shuffle.partitions=[num_tasks];�.
set spark.sql.shuffle.partitions= 1;
set spark.default.parallelism = 1;
set spark.sql.files.maxPartitionBytes = 1073741824; -- The maximum number of bytes to pack o a single partition when reading files.

Snippet

See doc ref

• spark-sql

 
SET -v

• Scala

sparkSession.sql("SET -v").show(numRows = 200, truncate = false)

• Java

sparkSession.sql("SET -v").show(200, false);

• Python

sparkSession.sql("SET -v").show(n=200, truncate=False)

• R

sparkR.session()
properties <- sql("SET -v")
showDF(properties, numRows = 200, truncate = FALSE)

List