Kafka - Docker Single Node (Multiple Service Broker + Zookeeper)

About

Docker Single Node step by step tutorial adapted from the Quickstart documentation.

Made:

1. on Windows 7
2. with Git Bash for Windows as console
3. with Confluent docker image version 3.3.0
4. with docker compose

docker-compose version

docker-compose version 1.16.1, build 6d1ac219
docker-py version: 2.5.1
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016

Articles Related

Step by Step

Get the docker-compose.yml file

• With git

git clone https://github.com/confluentinc/cp-docker-images.git
cd cp-docker-images/examples/kafka-single-node
cat docker-compose.yml

• or download it. https://github.com/confluentinc/cp-docker-images/blob/3.3.x/examples/kafka-single-node/docker-compose.yml

Modification of the docker-compose.yml file

Docker - docker-compose.yml need to be modified to:

• have the name of the host machine. Ie default in the extra_hosts section. See confluentinc/cp-docker-images/issues/261
• set the version of the image

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:3.3.0
    network_mode: host
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"
      - "default:127.0.0.1"
  kafka:
    image: confluentinc/cp-kafka:3.3.0
    network_mode: host
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: localhost:32181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    extra_hosts:
      - "moby:127.0.0.1"
      - "default:127.0.0.1"

Start the services

docker-compose up -d

Starting kafkasinglenode_zookeeper_1 ...
Starting kafkasinglenode_zookeeper_1 ... done
Starting kafkasinglenode_kafka_1 ...
Starting kafkasinglenode_kafka_1 ... done

• Up ? If the state is not Up, rerun the docker-compose up -d command.

docker-compose ps

Name                        Command            State   Ports
-----------------------------------------------------------------------
kafkasinglenode_kafka_1       /etc/confluent/docker/run   Up
kafkasinglenode_zookeeper_1   /etc/confluent/docker/run   Up

• Log Check Healthy Zookeeper

docker-compose logs zookeeper | grep -i binding

zookeeper_1  | [2017-10-12 13:17:35,956] INFO binding to port 0.0.0.0/0.0.0.0:32181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
zookeeper_1  | [2017-10-12 13:19:03,213] INFO binding to port 0.0.0.0/0.0.0.0:32181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

• Log Check Healthy Kafka

docker-compose logs kafka | grep -i started

kafka_1      | [2017-10-12 13:20:31,103] INFO [Socket Server on Broker 1], Started 1 acceptor threads (kafka.network.SocketServer)
kafka_1      | [2017-10-12 13:20:31,353] INFO [Replica state machine on controller 1]: Started replica state machine with initial state -> Map() (kafka.controller.ReplicaStateMachine)
kafka_1      | [2017-10-12 13:20:31,355] INFO [Partition state machine on Controller 1]: Started partition state machine with initial state -> Map() (kafka.controller.PartitionStateMachine)
kafka_1      | [2017-10-12 13:20:31,490] INFO [Kafka Server 1], started (kafka.server.KafkaServer)

Produce and Consume

• producer

docker-compose exec kafka  bash -c "seq 42 | kafka-console-producer --request-required-acks 1 --broker-list localhost:29092 --topic foo && echo 'Produced 42 messages.'"

Produced 42 messages.

• consumer

docker-compose exec kafka  \
  kafka-console-consumer --bootstrap-server localhost:29092 --topic foo --new-consumer --from-beginning --max-messages 42

...
36                                 
37                                 
38                                 
39                                 
40                                 
41                                 
42                                 
Processed a total of 42 messages