Kafka - Connect
About
Kafka Connect is a framework service based around connector to move data into and out of Kafka.
The service is a worker. By starting a worker, you are starting connect.
The management is done through the rest api and the connect name is often used to describe its rest api.
- Data sources: Kafka Connect - Source Connector (Producer/Write)
Articles Related
Concept
Connectors and tasks are logical units of work:
- Connectors – the high level abstraction that coordinates data streaming by managing tasks
- Tasks – the implementation of how data is copied to or from Kafka
Runtime:
- Workers – the running processes that execute connectors and tasks
Transform:
- Converters – Read/write implementation of a data format
- Transforms – simple logic to alter each message produced by or sent to a connector
Management
Start
Kafka Connect has only one hard prerequisite in order to get started: a set of Kafka brokers because it use it as a back-end database to store connector and task configurations, offsets, and status in internal Kafka topics
Connect (and therefore a worker) can be started
- as a standalone process for testing, one-off jobs and single agent (such as sending logs from webservers to Kafka)
- or as a distributed, scalable, fault tolerant service