File sink connector Adapted from Quickstart kafka connect
Normally, this steps follow the file-source demo and you have already a started machine called connect-demo.
If this is not the case, start it and open a shell:
docker start connect-demo
docker exec -it connect-demo //bin//bash
Verify that you have a connect-test topic
kafka-topics --list --zookeeper localhost:2181 | grep connect-test
connect-test
cat /etc/kafka/connect-file-sink.properties
# User defined name for the connector instance
# overwritten at the command line
name=local-file-sink
# Name of the connector class to be run
connector.class=FileStreamSink
# Max number of tasks to spawn for this connector instance
tasks.max=1
# Output file name relative to worker's current working directory
# This is the only property specific to the FileStreamSink connector
file=test.sink.txt
# Comma separate input topic list
topics=connect-test
confluent load file-sink -d /etc/kafka/connect-file-sink.properties
{
"name": "file-sink",
"config": {
"connector.class": "FileStreamSink",
"tasks.max": "1",
"file": "test.sink.txt",
"topics": "connect-test",
"name": "file-sink"
},
"tasks": []
}
confluent status file-sink
{
"name": "file-sink",
"connector": {
"state": "RUNNING",
"worker_id": "172.17.0.2:8083"
},
"tasks": [
{
"state": "RUNNING",
"id": 0,
"worker_id": "172.17.0.2:8083"
}
]
}
tail -f test.sink.txt
log line 1 for nico demo
log line 2 for nico demo
log line 3 for nico demo
log line 4 for nico demo
log line 5 for nico demo
log line 6 for nico demo
docker exec -it connect-demo //bin//
for i in {7..9}; do echo "log line $i for nico demo"; done > test.txtbash
....
log line 7 for nico demo
log line 8 for nico demo
log line 9 for nico demo