Kafka Connect - Standalone Mode

About

The runtime standalone mode of connect when running/starting a worker

Standalone mode is best suited for:

  • testing,
  • one-off jobs
  • or single agent (such as sending logs from webservers to Kafka)

You can still test a distributed mode locally by setting a different rest port. See Kafka Connect - Distributed Worker

Management

Start

To run a worker in standalone:

./bin/connect-standalone [-daemon] worker.properties connector1.properties [connector2.properties connector3.properties ...]

# Example
./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties ./etc/kafka-connect-jdbc/source-quickstart-sqlite.properties

where:

Users may optionally provide connector configurations at the command line as only a single worker instance exists and no coordination is required in standalone mode.

Log

When started as a deamon, the log is located to /logs/connectStandalone.out

  • Check the log
cat /logs/connectStandalone.out | grep -i "finished"
[2017-10-18 12:43:26,306] INFO Finished creating connector test-source-sqlite-jdbc-autoincrement (org.apache.kafka.connect.runtime.Worker:225)
[2017-10-18 12:43:26,425] INFO Source task WorkerSourceTask{id=test-source-sqlite-jdbc-autoincrement-0} finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:143)

Configuration

Worker

Kafka Connect - Worker in standalone mode.

See Work Config Reference If you run multiple standalone instances on the same host, there are a couple of settings that must be unique between each instance:

  • offset.storage.file.filename - storage for connector offsets, which are stored on the local filesystem in standalone mode; using the same file will lead to offset data being deleted or overwritten with different values
  • rest.port - the port the REST interface listens on for HTTP requests

Connector

name=local-file-sink
connector.class=FileStreamSinkConnector
tasks.max=1
file=test.sink.txt
topics=connect-test

Powered by ComboStrap