Table of Contents

Kafka - Installation Standalone / Open Source (Single Broker)

About

This page shows you how to install kafka from the open source package with a single broker (a single node)

Prerequisites

Kafka is working with zookeeper to store its data. A zookeeper server must be running before starting Kafka. See Zookeeper - Installation

You can see the Zookeeper version dependency in the kafka_home/libs/zookeeper-x.x.x.jar where x.x.x is the version.

Steps

The installation directories

KAFKA_BASE_DIR=/opt
mkdir -p ${KAFKA_BASE_DIR}

KAFKA_HOME_DIR=${KAFKA_BASE_DIR}/kafka

Download and unzip

KAFKA_VERSION=2.1.1
KAFKA_SCALA_VERSION=2.11
KAFKA_PACKAGE_NAME=kafka_${KAFKA_VERSION}-${KAFKA_SCALA_VERSION}
KAFKA_PACKAGE_URL=https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/${KAFKA_PACKAGE_NAME}.tgz

KAFKA_DOWNLOAD_DIR=$HOME/.kafka/download
KAFKA_DOWNLOAD_FILE="$KAFKA_DOWNLOAD_DIR/$(basename $KAFKA_PACKAGE_URL)"
curl "$KAFKA_PACKAGE_URL" > "${KAFKA_DOWNLOAD_FILE}.tmp"
mv "${KAFKA_DOWNLOAD_FILE}.tmp" "${KAFKA_DOWNLOAD_FILE}"
tar -xf "${KAFKA_DOWNLOAD_FILE}" -C $KAFKA_DOWNLOAD_DIR

Install

mv "$KAFKA_PACKAGE_NAME" "$KAFKA_HOME_DIR" 

# have to use SIGTERM since nohup on appears to ignore SIGINT
# and Kafka switched to SIGINT in KAFKA-1031.
sed -i.bak 's/SIGINT/SIGTERM/g' $KAFKA_HOME_DIR/bin/kafka-server-stop.sh
# in order to simplify the wikipedia-stats example job, set topic to have just 1 partition by default
sed -i.bak 's/^num\.partitions *=.*/num.partitions=1/' $KAFKA_HOME_DIR/config/server.properties

Listener changes for a local development

Properties change in the KAFKA_HOME_DIR/config/server.properties for a local development (change via sed ) on the listeners properties. 

sed -i "s/\#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/0.0.0.0:9092/" $KAFKA_HOME_DIR/config/server.properties 

sed -i "s/\#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=PLAINTEXT:\/\/localhost:9092/" $KAFKA_HOME_DIR/config/server.properties

listeners property is a comma-separated list of URIs) where to listen.

The scheme (PLAINTEXT, …) is called a security protocol.

Example:

Services

Start

mkdir -p $KAFKA_HOME_DIR/logs
cd $KAFKA_HOME_DIR
nohup bin/kafka-server-start.sh config/server.properties > logs/kafka.log 2>&1 &
cd - > /dev/null
wait_for_service "kafka" 9092

where:

More … Multibrocker

See http://kafka.apache.org/documentation.html#quickstart_multibroker

Documentation / Reference