Kafka Connect - Connector Plugin

About

Connector is a component of the connect framework that coordinates data streaming by managing tasks

A connector instance is a logical job. Each connector instance coordinates a set of tasks that actually copy the data.

A connector is run by a worker.

Management

See:

The easiest way to create, configure, and manage connectors is with Confluent Control Center.

State

The following states are possible for a connector or one of its tasks:

  • UNASSIGNED: The connector/task has not yet been assigned to a worker.
  • RUNNING: The connector/task is running.
  • PAUSED: The connector/task has been administratively paused.
  • FAILED: The connector/task has failed (usually by raising an exception, which is reported in the status output).

Concept

Type

  • Kafka Connect - Sink Connector (Read)
  • Kafka Connect - Source Connector (Producer/Write)

Built-in consumer, producer

Kafka Connect uses the standard Java producer and consumers to communicate with Kafka.

All producer configs and consumer configs can be overridden by prefixing them with producer. or consumer., respectively.

List

Actual Connector

See list connector available for a worker (in distributed mode).

Externally Available

Dev Guide / Build

mvn archetype:generate -DarchetypeGroupId=com.github.jcustenborder.kafka.connect -DarchetypeArtifactId=kafka-connect-quickstart -DarchetypeVersion=0.11.0.0-cp1

Configuration

Connector configuration file are given:

Common properties:

  • name - Unique name for the connector. Attempting to register again with the same name will fail.
  • connector.class - The Java class for the connector
  • tasks.max - The maximum number of tasks that should be created for this connector. The connector may create fewer tasks if it cannot achieve this level of parallelism.
  • key.converter and value.converter - (optional) to override the default converter configuration

Sink connectors:

  • topics - A list of topics to use as input for this connector

Offset

Even if you delete and recreate the connector, if the connector has the same name it will retain the same offsets previously stored.

Documentation / Reference


Powered by ComboStrap