Kafka Connect - File Sink connector

1 - About

3 - Steps

3.1 - Start docker

Normally, this steps follow the file-source demo and you have already a started machine called connect-demo.

If this is not the case, start it and open a shell:

docker start connect-demo
docker exec -it connect-demo //bin//bash

Verify that you have a connect-test topic

kafka-topics --list --zookeeper localhost:2181 | grep connect-test


3.2 - Load the sink connector

  • Check the configuration file

cat /etc/kafka/connect-file-sink.properties

# User defined name for the connector instance
# overwritten at the command line
# Name of the connector class to be run
# Max number of tasks to spawn for this connector instance
# Output file name relative to worker's current working directory
# This is the only property specific to the FileStreamSink connector
# Comma separate input topic list

  • Load it

confluent load file-sink -d /etc/kafka/connect-file-sink.properties

  "name": "file-sink",                  
  "config": {                           
    "connector.class": "FileStreamSink",
    "tasks.max": "1",                   
    "file": "test.sink.txt",            
    "topics": "connect-test",           
    "name": "file-sink"                 
  "tasks": []                           

  • Status

confluent status file-sink

  "name": "file-sink",
  "connector": {
    "state": "RUNNING",
    "worker_id": ""
  "tasks": [
      "state": "RUNNING",
      "id": 0,
      "worker_id": ""

3.3 - Check the create file

tail -f test.sink.txt

log line 1 for nico demo
log line 2 for nico demo
log line 3 for nico demo
log line 4 for nico demo
log line 5 for nico demo
log line 6 for nico demo

3.4 - Add record to the source

  • Open a new bash and add line to the file

docker exec -it connect-demo //bin//
for i in {7..9}; do echo "log line $i for nico demo"; done > test.txtbash

  • Check the console with the tail command, you must see the new lines coming in.

log line 7 for nico demo
log line 8 for nico demo
log line 9 for nico demo

Data Science
Data Analysis
Data Science
Linear Algebra Mathematics

Powered by ComboStrap