Kafka - Stream Word Count demo

1 - About

Stream Word count demo adapted from the stream quickstart.

3 - Prerequisites

  • Install docker: Docker - Installation
  • Map the hostname zookeeper and broker to your docker host ip in your host file. Example:

192.168.99.100   broker
192.168.99.100   zookeeper

4 - Steps

4.1 - Start the services

  • Start the kafka server and zookeeper

cd /kafka/docker/stream
export COMPOSE_PROJECT_NAME="stream-demo"
docker-compose up -d


Creating network "streamdemo_default" with the default driver
Creating streamdemo_zookeeper_1 ...
Creating streamdemo_zookeeper_1 ... done
Creating streamdemo_broker_1 ...
Creating streamdemo_broker_1 ... done

  • Service status

docker-compose ps


Name                     Command            State                     Ports
-------------------------------------------------------------------------------------------------------
streamdemo_broker_1      /etc/confluent/docker/run   Up      0.0.0.0:9092->9092/tcp
streamdemo_zookeeper_1   /etc/confluent/docker/run   Up      0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

4.2 - Create a topic


# Enter in the broker container and start bash
docker-compose exec broker bash
# Create the topic
kafka-topics --create \
    --zookeeper zookeeper:2181 \
    --replication-factor 1 \
    --partitions 1 \
    --topic streams-plaintext-input


Created topic "streams-plaintext-input".

  • Create the output topic

kafka-topics --create \
    --zookeeper zookeeper:2181 \
    --replication-factor 1 \
    --partitions 1 \
    --topic streams-wordcount-output \
    --config cleanup.policy=compact

Created topic "streams-wordcount-output".

  • Describe them

kafka-topics \
    --zookeeper zookeeper:2181 \
    --describe

Topic:streams-plaintext-input   PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: streams-plaintext-input  Partition: 0    Leader: 1       Replicas: 1     Isr: 1

Topic:streams-wordcount-output  PartitionCount:1        ReplicationFactor:1     Configs:cleanup.policy=compact
        Topic: streams-wordcount-output Partition: 0    Leader: 1       Replicas: 1     Isr: 1

4.3 - Code

4.4 - Packaging


java -cp ./target/kafka-streams-0.1-standalone.jar com.gerardnico.kafka.demo.WordCountDemo


Data Science
Data Analysis
Statistics
Data Science
Linear Algebra Mathematics
Trigonometry

Powered by ComboStrap