Spark DataSet - (Object) Encoder

About

To define a dataset Object, an encoder is required.

It is used to tell Spark to generate code at runtime to serialize the object into a binary structure.

This binary structure often has much lower memory footprint as well as are optimized for efficiency in data processing (e.g. in a columnar format).

The encoder maps the object type T

to Spark's internal type system.

Example

Given a class Person with two fields, name (string) and age (int)

Dataset<Person> people = spark.read().parquet("...").as(Encoders.bean(Person.class)); 

Management

Specification

  • Scala Encoders are generally created automatically through implicits from a SparkSession, or can be explicitly created by calling static methods on Encoders.
import spark.implicits._
val ds = Seq(1, 2, 3).toDS() // implicitly provided (spark.implicits.newIntEncoder)
  • Java: Encoders are specified by calling static methods on Encoders.
List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());

Construction

Tuples

Encoders can be composed into tuples (logical row):

Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

Java Beans

constructed from Java Beans:

Encoders.bean(MyClass.class);

Documentation / Reference


Powered by ComboStrap