Kafka - Connect

About

Kafka Connect is a framework service based around connector to move data into and out of Kafka.

The service is a worker. By starting a worker, you are starting connect.

The management is done through the rest api and the connect name is often used to describe its rest api.

Concept

Connectors and tasks are logical units of work:

  • Connectors – the high level abstraction that coordinates data streaming by managing tasks
  • Tasks – the implementation of how data is copied to or from Kafka

Runtime:

  • Workers – the running processes that execute connectors and tasks

Transform:

  • Converters – Read/write implementation of a data format
  • Transforms – simple logic to alter each message produced by or sent to a connector

Converter Basics

Management

Start

Kafka Connect has only one hard prerequisite in order to get started: a set of Kafka brokers because it use it as a back-end database to store connector and task configurations, offsets, and status in internal Kafka topics

Connect (and therefore a worker) can be started

  • as a standalone process for testing, one-off jobs and single agent (such as sending logs from webservers to Kafka)
  • or as a distributed, scalable, fault tolerant service

Documentation / Reference

Task Runner