Kafka Connect - File Sink connector

1 - About

3 - Steps

3.1 - Start docker

Normally, this steps follow the file-source demo and you have already a started machine called connect-demo.

If this is not the case, start it and open a shell:


docker start connect-demo
docker exec -it connect-demo //bin//bash

Verify that you have a connect-test topic


kafka-topics --list --zookeeper localhost:2181 | grep connect-test


connect-test

3.2 - Load the sink connector

  • Check the configuration file

cat /etc/kafka/connect-file-sink.properties


# User defined name for the connector instance
# overwritten at the command line
name=local-file-sink
# Name of the connector class to be run
connector.class=FileStreamSink
# Max number of tasks to spawn for this connector instance
tasks.max=1
# Output file name relative to worker's current working directory
# This is the only property specific to the FileStreamSink connector
file=test.sink.txt
# Comma separate input topic list
topics=connect-test

  • Load it

confluent load file-sink -d /etc/kafka/connect-file-sink.properties


{                                       
  "name": "file-sink",                  
  "config": {                           
    "connector.class": "FileStreamSink",
    "tasks.max": "1",                   
    "file": "test.sink.txt",            
    "topics": "connect-test",           
    "name": "file-sink"                 
  },                                    
  "tasks": []                           
}                                       

  • Status

confluent status file-sink


{
  "name": "file-sink",
  "connector": {
    "state": "RUNNING",
    "worker_id": "172.17.0.2:8083"
  },
  "tasks": [
    {
      "state": "RUNNING",
      "id": 0,
      "worker_id": "172.17.0.2:8083"
    }
  ]
}

3.3 - Check the create file


tail -f test.sink.txt


log line 1 for nico demo
log line 2 for nico demo
log line 3 for nico demo
log line 4 for nico demo
log line 5 for nico demo
log line 6 for nico demo

3.4 - Add record to the source

  • Open a new bash and add line to the file

docker exec -it connect-demo //bin//
for i in {7..9}; do echo "log line $i for nico demo"; done > test.txtbash

  • Check the console with the tail command, you must see the new lines coming in.

....
log line 7 for nico demo
log line 8 for nico demo
log line 9 for nico demo


Data Science
Data Analysis
Statistics
Data Science
Linear Algebra Mathematics
Trigonometry

Powered by ComboStrap