Kafka Connect - Standalone Mode

Converter Basics

Kafka Connect - Standalone Mode

About

The runtime standalone mode of connect when running/starting a worker

Standalone mode is best suited for:

  • testing,
  • one-off jobs
  • or single agent (such as sending logs from webservers to Kafka)

You can still test a distributed mode locally by setting a different rest port. See Kafka Connect - Distributed Worker

Management

Start

To run a worker in standalone: 

./bin/connect-standalone [-daemon] worker.properties connector1.properties [connector2.properties connector3.properties ...]

# Example
./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties ./etc/kafka-connect-jdbc/source-quickstart-sqlite.properties

where:

Users may optionally provide connector configurations at the command line as only a single worker instance exists and no coordination is required in standalone mode.

Log

When started as a deamon, the log is located to /logs/connectStandalone.out

  • Check the log
cat /logs/connectStandalone.out | grep -i "finished"

[2017-10-18 12:43:26,306] INFO Finished creating connector test-source-sqlite-jdbc-autoincrement (org.apache.kafka.connect.runtime.Worker:225)
[2017-10-18 12:43:26,425] INFO Source task WorkerSourceTask{id=test-source-sqlite-jdbc-autoincrement-0} finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:143)

Configuration

Worker

Kafka Connect - Worker in standalone mode.

See Work Config Reference If you run multiple standalone instances on the same host, there are a couple of settings that must be unique between each instance:

  • offset.storage.file.filename - storage for connector offsets, which are stored on the local filesystem in standalone mode; using the same file will lead to offset data being deleted or overwritten with different values
  • rest.port - the port the REST interface listens on for HTTP requests

Connector

name=local-file-sink
connector.class=FileStreamSinkConnector
tasks.max=1
file=test.sink.txt
topics=connect-test




Discover More
Converter Basics
Kafka - Connect

Kafka Connect is a framework service based around connector to move data into and out of Kafka. The service is a worker. By starting a worker, you are starting connect. The management is done through...
Kafka Commit Log Messaging Process
Kafka - Tutorials

A list of tutorials that I made on windows. They were my path to learn Kafka confluent In this tutorial: you start all services in a docker container you write and read data into a topic...
Converter Basics
Kafka Connect - Connector Plugin

Connector is a component of the connect framework that coordinates data streaming by managing tasks A connector instance is a logical job. Each connector instance coordinates a set of tasks that actually...
Converter Basics
Kafka Connect - Worker

A worker is a connect component. It is the running process (JVM processes) that execute tasks of a connector. A worker may run several connectors. There is two types of runtime workers: standalone:...
Kafka Commit Log Messaging Process
Sqlite Standalone

Windows where: connection.url is the JDBC URL mode indicates how we want to query the data. incrementing means that each query for new data will only return rows with IDs larger than the maximum...



Share this page:
Follow us:
Task Runner