Kafka Connect - File Sink connector

Kafka Commit Log Messaging Process

About

File sink connector Adapted from Quickstart kafka connect

Steps

Start docker

Normally, this steps follow the file-source demo and you have already a started machine called connect-demo.

If this is not the case, start it and open a shell:

docker start connect-demo
docker exec -it connect-demo //bin//bash

Verify that you have a connect-test topic

kafka-topics --list --zookeeper localhost:2181 | grep connect-test
connect-test

Load the sink connector

  • Check the configuration file
cat /etc/kafka/connect-file-sink.properties
# User defined name for the connector instance
# overwritten at the command line
name=local-file-sink
# Name of the connector class to be run
connector.class=FileStreamSink
# Max number of tasks to spawn for this connector instance
tasks.max=1
# Output file name relative to worker's current working directory
# This is the only property specific to the FileStreamSink connector
file=test.sink.txt
# Comma separate input topic list
topics=connect-test

  • Load it
confluent load file-sink -d /etc/kafka/connect-file-sink.properties
{                                       
  "name": "file-sink",                  
  "config": {                           
    "connector.class": "FileStreamSink",
    "tasks.max": "1",                   
    "file": "test.sink.txt",            
    "topics": "connect-test",           
    "name": "file-sink"                 
  },                                    
  "tasks": []                           
}                                       

  • Status
confluent status file-sink
{
  "name": "file-sink",
  "connector": {
    "state": "RUNNING",
    "worker_id": "172.17.0.2:8083"
  },
  "tasks": [
    {
      "state": "RUNNING",
      "id": 0,
      "worker_id": "172.17.0.2:8083"
    }
  ]
}

Check the create file

tail -f test.sink.txt
log line 1 for nico demo
log line 2 for nico demo
log line 3 for nico demo
log line 4 for nico demo
log line 5 for nico demo
log line 6 for nico demo

Add record to the source

  • Open a new bash and add line to the file
docker exec -it connect-demo //bin//
for i in {7..9}; do echo "log line $i for nico demo"; done > test.txtbash
  • Check the console with the tail command, you must see the new lines coming in.
....
log line 7 for nico demo
log line 8 for nico demo
log line 9 for nico demo





Discover More
Kafka Commit Log Messaging Process
Kafka Connect - File

File connector Built-in Connector Both source and sink connectors can track offsets, so you can start and stop the process any number of times and add more data to the input file and both...



Share this page:
Follow us:
Task Runner