Kafka Connect - Worker

Converter Basics

About

A worker is a connect component. It is the running process (JVM processes) that execute tasks of a connector.

A worker may run several connectors.

There is two types of runtime workers:

Management

Start

Configuration

Documentation / Reference




Discover More
Kafka Commit Log Messaging Process
Kafka - Architecture

into secondary indexes like Elasticsearch into batch systems such as Hadoop for offline analysis. Workers lend themselves well to being run in containers in managed environments such as Kubernetes,...
Converter Basics
Kafka - Connect

Kafka Connect is a framework service based around connector to move data into and out of Kafka. The service is a worker. By starting a worker, you are starting connect. The management is done through...
Kafka Commit Log Messaging Process
Kafka - Tutorials

A list of tutorials that I made on windows. They were my path to learn Kafka confluent In this tutorial: you start all services in a docker container you write and read data into a topic...
Converter Basics
Kafka Connect - Connector Plugin

Connector is a component of the connect framework that coordinates data streaming by managing tasks A connector instance is a logical job. Each connector instance coordinates a set of tasks that actually...
Converter Basics
Kafka Connect - Converter (Read/Write - I/O)

A converter is a connect concept. It's the code used to persist data from a Connector. Converters are decoupled from connectors themselves to allow for reuse. For example, using the same Avro...
Worker Model Basics
Kafka Connect - Distributed Worker

The runtime distributed mode of connect when running/starting a worker See where: worker.properties is the configuration file New workers will either start a new group or join an existing...
Converter Basics
Kafka Connect - Plugin

A Kafka Connect plugin is simply a set of JAR files where Kafka Connect can find an implementation of one or more: connectors, transforms, and/or converters. Kafka Connect isolates each...
Converter Basics
Kafka Connect - REST API

The Connect Rest api is the management interface for the connect service. Unlike many other systems, all nodes in Kafka Connect can respond to REST requests, including creating, listing, modifying, and...
Converter Basics
Kafka Connect - Standalone Mode

The runtime standalone mode of connect when running/starting a worker Standalone mode is best suited for: testing, one-off jobs or single agent (such as sending logs from webservers to Kafka)...
Converter Basics
Kafka Connect - Task

Task is a connector implementation on how data is copied to or from Kafka These tasks have no state stored within them. Task state is stored in Kafka in special topics and managed by the associated...



Share this page:
Follow us:
Task Runner