Kafka Connect - File Sink connector



Start docker

Normally, this steps follow the file-source demo and you have already a started machine called connect-demo.

If this is not the case, start it and open a shell:

docker start connect-demo
docker exec -it connect-demo //bin//bash

Verify that you have a connect-test topic

kafka-topics --list --zookeeper localhost:2181 | grep connect-test

Load the sink connector

  • Check the configuration file
cat /etc/kafka/connect-file-sink.properties
# User defined name for the connector instance
# overwritten at the command line
# Name of the connector class to be run
# Max number of tasks to spawn for this connector instance
# Output file name relative to worker's current working directory
# This is the only property specific to the FileStreamSink connector
# Comma separate input topic list
  • Load it
confluent load file-sink -d /etc/kafka/connect-file-sink.properties
  "name": "file-sink",                  
  "config": {                           
    "connector.class": "FileStreamSink",
    "tasks.max": "1",                   
    "file": "test.sink.txt",            
    "topics": "connect-test",           
    "name": "file-sink"                 
  "tasks": []                           
  • Status
confluent status file-sink
  "name": "file-sink",
  "connector": {
    "state": "RUNNING",
    "worker_id": ""
  "tasks": [
      "state": "RUNNING",
      "id": 0,
      "worker_id": ""

Check the create file

tail -f test.sink.txt
log line 1 for nico demo
log line 2 for nico demo
log line 3 for nico demo
log line 4 for nico demo
log line 5 for nico demo
log line 6 for nico demo

Add record to the source

  • Open a new bash and add line to the file
docker exec -it connect-demo //bin//
for i in {7..9}; do echo "log line $i for nico demo"; done > test.txtbash
  • Check the console with the tail command, you must see the new lines coming in.
log line 7 for nico demo
log line 8 for nico demo
log line 9 for nico demo

Powered by ComboStrap