A connector instance is a logical job. Each connector instance coordinates a set of tasks that actually copy the data.
A connector is run by a worker.
The following states are possible for a connector or one of its tasks:
- UNASSIGNED: The connector/task has not yet been assigned to a worker.
- RUNNING: The connector/task is running.
- PAUSED: The connector/task has been administratively paused.
- FAILED: The connector/task has failed (usually by raising an exception, which is reported in the status output).
Built-in consumer, producer
Kafka Connect uses the standard Java producer and consumers to communicate with Kafka.
See list connector available for a worker (in distributed mode).
What connectors can I install ?
Dev Guide / Build
mvn archetype:generate -DarchetypeGroupId=com.github.jcustenborder.kafka.connect -DarchetypeArtifactId=kafka-connect-quickstart -DarchetypeVersion=0.11.0.0-cp1
Connector configuration file are given:
- when a worker is started in standalone mode
- via Rest Api for the distributed mode. See Kafka Connect - Distributed Worker. The Connector configuration is persisted into Kafka. Every time you start a worker, it will start all the connectors that were running when it was stopped. The configuration data will be persisted in a the internal topic connect-configs.
- name - Unique name for the connector. Attempting to register again with the same name will fail.
- connector.class - The Java class for the connector
- tasks.max - The maximum number of tasks that should be created for this connector. The connector may create fewer tasks if it cannot achieve this level of parallelism.
- key.converter and value.converter - (optional) to override the default converter configuration
- topics - A list of topics to use as input for this connector
Even if you delete and recreate the connector, if the connector has the same name it will retain the same offsets previously stored.