Kafka Connect - Converter (Read/Write - I/O)

Converter Basics

About

A converter is a connect concept.

It's the code used to persist data from a Connector.

Interface Read/Write

Converters are decoupled from connectors themselves to allow for reuse.

For example, using the same Avro converter

  • the JDBC Source Connector can write Avro data to Kafka
  • and the HDFS Sink Connector can read Avro data from Kafka.

Converter Basics

Management

List

The converter properties in a worker configuration file specify the converters.

Type Encode / Decode
key.converter key
value.converter value
internal.converter internal storage topics

Configuration

Each converter implementation will have its own associated configuration requirements.

To pass configuration parameters to key and value converters, prefix them with key.converter. or value.converter.

Example:

  • An AvroConverter bundled with the Schema Registry
key.converter=io.confluent.connect.avro.AvroConverter
# required when bundled with the Schema Registry 
key.converter.schema.registry.url=http://localhost:8081
  • JsonConverter without schemas
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false

Built-in

converters for:

  • Avro, (Default/Recommended)
  • JSON,
  • and String initially.

Documentation / Reference





Discover More
Kafka Commit Log Messaging Process
Kafka - Avro Converter

in Kafka. The Avro converter is normally used with the Schema Registry in order to properly lookup the schema for the Avro data. ...
Converter Basics
Kafka - Connect

Kafka Connect is a framework service based around connector to move data into and out of Kafka. The service is a worker. By starting a worker, you are starting connect. The management is done through...
Kafka Commit Log Messaging Process
Kafka - JSON Converter

JSON converter The value.converter.schemas.enable and/or key.converter.schemas.enable must be set to false to have the JSON contain only they value and key, respectively, without the schemas.
Converter Basics
Kafka Connect - Connector Plugin

Connector is a component of the connect framework that coordinates data streaming by managing tasks A connector instance is a logical job. Each connector instance coordinates a set of tasks that actually...
Kafka Commit Log Messaging Process
Kafka Connect - File Source connector

Reading File with connect. Adapted from Quickstart kafka connect distributed mode Start the demo docker image...
Kafka Commit Log Messaging Process
Kafka Connect - JDBC Connector

Configuration file: confluentinc/kafka-connect-jdbc/releases or the confluent repo ...
Converter Basics
Kafka Connect - Plugin

A Kafka Connect plugin is simply a set of JAR files where Kafka Connect can find an implementation of one or more: connectors, transforms, and/or converters. Kafka Connect isolates each...
Converter Basics
Kafka Connect - Storage Topics

The distributed workers are stateless. The data are stored within internal Kafka topics: connector configurations task configurations, offsets, and status Connect creates this topics...
Converter Basics
Kafka Connect - Transform (Single Message Transform - SMT)

Transform is a concept of connect to apply simple transformation. A transform is a simple function that accepts one record as input and outputs a modified record. Transform are chained. In case of...



Share this page:
Follow us:
Task Runner