Kafka - Connect

1 - About

Kafka Connect is a framework service based around connector to move data into and out of Kafka.

The service is a worker. By starting a worker, you are starting connect.

The management is done through the rest api and the connect name is often used to describe its rest api.

  • Data sources: Kafka Connect - Source Connector (Producer/Write)
  • Target: Kafka Connect - Sink Connector (Read)

3 - Concept

Connectors and tasks are logical units of work:

  • Connectors – the high level abstraction that coordinates data streaming by managing tasks
  • Tasks – the implementation of how data is copied to or from Kafka


  • Workers – the running processes that execute connectors and tasks


  • Converters – Read/write implementation of a data format
  • Transforms – simple logic to alter each message produced by or sent to a connector

4 - Management

4.1 - Start

Kafka Connect has only one hard prerequisite in order to get started: a set of Kafka brokers because it use it as a back-end database to store connector and task configurations, offsets, and status in internal Kafka topics

Connect (and therefore a worker) can be started

  • as a standalone process for testing, one-off jobs and single agent (such as sending logs from webservers to Kafka)
  • or as a distributed, scalable, fault tolerant service

5 - Documentation / Reference

Data Science
Data Analysis
Data Science
Linear Algebra Mathematics

Powered by ComboStrap