Kafka Connect - File Sink connector

Start docker

Normally, this steps follow the file-source demo and you have already a started machine called connect-demo.

If this is not the case, start it and open a shell:

docker start connect-demo
docker exec -it connect-demo //bin//bash

Verify that you have a connect-test topic

kafka-topics --list --zookeeper localhost:2181 | grep connect-test

connect-test

Load the sink connector

• Check the configuration file

cat /etc/kafka/connect-file-sink.properties

# User defined name for the connector instance
# overwritten at the command line
name=local-file-sink
# Name of the connector class to be run
connector.class=FileStreamSink
# Max number of tasks to spawn for this connector instance
tasks.max=1
# Output file name relative to worker's current working directory
# This is the only property specific to the FileStreamSink connector
file=test.sink.txt
# Comma separate input topic list
topics=connect-test

• Load it

confluent load file-sink -d /etc/kafka/connect-file-sink.properties

{                                       
  "name": "file-sink",                  
  "config": {                           
    "connector.class": "FileStreamSink",
    "tasks.max": "1",                   
    "file": "test.sink.txt",            
    "topics": "connect-test",           
    "name": "file-sink"                 
  },                                    
  "tasks": []                           
}                                       

• Status

confluent status file-sink

{
  "name": "file-sink",
  "connector": {
    "state": "RUNNING",
    "worker_id": "172.17.0.2:8083"
  },
  "tasks": [
    {
      "state": "RUNNING",
      "id": 0,
      "worker_id": "172.17.0.2:8083"
    }
  ]
}

Check the create file

tail -f test.sink.txt

log line 1 for nico demo
log line 2 for nico demo
log line 3 for nico demo
log line 4 for nico demo
log line 5 for nico demo
log line 6 for nico demo

Add record to the source

• Open a new bash and add line to the file

docker exec -it connect-demo //bin//
for i in {7..9}; do echo "log line $i for nico demo"; done > test.txtbash

• Check the console with the tail command, you must see the new lines coming in.

....
log line 7 for nico demo
log line 8 for nico demo
log line 9 for nico demo