Kafka - Stream Word Count demo
Table of Contents
About
Stream Word count demo adapted from the stream quickstart.
Articles Related
Prerequisites
- Install docker: Docker - Installation
- Map the hostname zookeeper and broker to your docker host ip in your host file. Example:
192.168.99.100 broker
192.168.99.100 zookeeper
Steps
Start the services
- Start the kafka server and zookeeper
cd /kafka/docker/stream
export COMPOSE_PROJECT_NAME="stream-demo"
docker-compose up -d
Creating network "streamdemo_default" with the default driver
Creating streamdemo_zookeeper_1 ...
Creating streamdemo_zookeeper_1 ... done
Creating streamdemo_broker_1 ...
Creating streamdemo_broker_1 ... done
- Service status
docker-compose ps
Name Command State Ports
-------------------------------------------------------------------------------------------------------
streamdemo_broker_1 /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp
streamdemo_zookeeper_1 /etc/confluent/docker/run Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp
Create a topic
- Create the input topic
# Enter in the broker container and start bash
docker-compose exec broker bash
# Create the topic
kafka-topics --create \
--zookeeper zookeeper:2181 \
--replication-factor 1 \
--partitions 1 \
--topic streams-plaintext-input
Created topic "streams-plaintext-input".
- Create the output topic
kafka-topics --create \
--zookeeper zookeeper:2181 \
--replication-factor 1 \
--partitions 1 \
--topic streams-wordcount-output \
--config cleanup.policy=compact
Created topic "streams-wordcount-output".
- Describe them
kafka-topics \
--zookeeper zookeeper:2181 \
--describe
Topic:streams-plaintext-input PartitionCount:1 ReplicationFactor:1 Configs:
Topic: streams-plaintext-input Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic:streams-wordcount-output PartitionCount:1 ReplicationFactor:1 Configs:cleanup.policy=compact
Topic: streams-wordcount-output Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Code
Packaging
java -cp ./target/kafka-streams-0.1-standalone.jar com.gerardnico.kafka.demo.WordCountDemo