Spark - Sql

About

This section is about:

• the SQL Grammar of Spark
• and the SQL Thrift Server.

SQL is an interface to the spark Spark Sql engine that supports:

• all existing Hive data formats,
• the hive syntax (user-defined functions - UDF),
• and the Hive metastore to get the metadata of the data stored in HDFS.

Spark SQL is the standard for SQL on Spark. Hive on Spark is similar to SparkSQL but aims to leverage existing investments in Hive (security, …) on Spark. They are two different components with two different code base.

Features:

• Runs SQL / HiveQL queries, optionally alongside or replacing existing Hive deployments
• Connect existing BI tools to Spark through JDBC

Articles Related

Documentation / Reference

• Spark SQL = distributed-sql-engine
• https://stackoverflow.com/questions/31611744/sparksql-vs-hive-on-spark-difference-and-pros-and-cons
• https://databricks.com/blog/2014/07/01/shark-spark-sql-hive-on-spark-and-the-future-of-sql-on-spark.html
• https://docs.azuredatabricks.net/spark/latest/spark-sql/index.html