Spark DataSet - Data Type

Card Puncher Data Processing

Spark DataSet - Data Type

About

Same as Spark SQL - Data Type

Coercion

Python

from pyspark.sql.functions import col , column
dataFrame = dataFrame.withColumn("price", col("price").cast("double"))
dataFrame = dataFrame.withColumn("time_at", col("time_at").cast("timestamp"))

List

Supported types are:

  • string,
  • boolean,
  • byte,
  • short,
  • int,
  • long,
  • float,
  • double,
  • decimal,
  • date,
  • timestamp





Discover More
Spark Sql Datatype
Spark SQL - Data Type

Sql Data Type in Spark. From a JDBC driver point of view



Share this page:
Follow us:
Task Runner