Spark DataSet - (Object) Encoder

1 - About

To define a dataset Object, an encoder is required.

It is used to tell Spark to generate code at runtime to serialize the object into a binary structure.

This binary structure often has much lower memory footprint as well as are optimized for efficiency in data processing (e.g. in a columnar format).

The encoder maps the object type T

to Spark's internal type system.

3 - Example

Given a class Person with two fields, name (string) and age (int)


Dataset<Person> people = spark.read().parquet("...").as(Encoders.bean(Person.class)); 

4 - Management

4.1 - Specification

  • Scala Encoders are generally created automatically through implicits from a SparkSession, or can be explicitly created by calling static methods on Encoders.

import spark.implicits._
val ds = Seq(1, 2, 3).toDS() // implicitly provided (spark.implicits.newIntEncoder)

  • Java: Encoders are specified by calling static methods on Encoders.

List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());

4.2 - Construction

4.2.1 - Tuples

Encoders can be composed into tuples (logical row):


Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

4.2.2 - Java Beans

constructed from Java Beans:


Encoders.bean(MyClass.class);

5 - Documentation / Reference


Data Science
Data Analysis
Statistics
Data Science
Linear Algebra Mathematics
Trigonometry

Powered by ComboStrap