About
This page shows you how to install kafka from the open source package with a single broker (a single node)
Articles Related
Prerequisites
Kafka is working with zookeeper to store its data. A zookeeper server must be running before starting Kafka. See Zookeeper - Installation
You can see the Zookeeper version dependency in the kafka_home/libs/zookeeper-x.x.x.jar where x.x.x is the version.
Steps
The installation directories
- The base installation directory
KAFKA_BASE_DIR=/opt
mkdir -p ${KAFKA_BASE_DIR}
- The home directory
KAFKA_HOME_DIR=${KAFKA_BASE_DIR}/kafka
Download and unzip
- Version and url
KAFKA_VERSION=2.1.1
KAFKA_SCALA_VERSION=2.11
KAFKA_PACKAGE_NAME=kafka_${KAFKA_VERSION}-${KAFKA_SCALA_VERSION}
KAFKA_PACKAGE_URL=https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/${KAFKA_PACKAGE_NAME}.tgz
- Download and unzip
KAFKA_DOWNLOAD_DIR=$HOME/.kafka/download
KAFKA_DOWNLOAD_FILE="$KAFKA_DOWNLOAD_DIR/$(basename $KAFKA_PACKAGE_URL)"
curl "$KAFKA_PACKAGE_URL" > "${KAFKA_DOWNLOAD_FILE}.tmp"
mv "${KAFKA_DOWNLOAD_FILE}.tmp" "${KAFKA_DOWNLOAD_FILE}"
tar -xf "${KAFKA_DOWNLOAD_FILE}" -C $KAFKA_DOWNLOAD_DIR
Install
mv "$KAFKA_PACKAGE_NAME" "$KAFKA_HOME_DIR"
- Properties change via sed
# have to use SIGTERM since nohup on appears to ignore SIGINT
# and Kafka switched to SIGINT in KAFKA-1031.
sed -i.bak 's/SIGINT/SIGTERM/g' $KAFKA_HOME_DIR/bin/kafka-server-stop.sh
# in order to simplify the wikipedia-stats example job, set topic to have just 1 partition by default
sed -i.bak 's/^num\.partitions *=.*/num.partitions=1/' $KAFKA_HOME_DIR/config/server.properties
Listener changes for a local development
Properties change in the KAFKA_HOME_DIR/config/server.properties for a local development (change via sed ) on the listeners properties.
- listeners=PLAINTEXT://0.0.0.0:9092 to listen on all interface
sed -i "s/\#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/0.0.0.0:9092/" $KAFKA_HOME_DIR/config/server.properties
- advertised.listeners=PLAINTEXT://localhost:9092 to redirect the client to localhost as FQDN
sed -i "s/\#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=PLAINTEXT:\/\/localhost:9092/" $KAFKA_HOME_DIR/config/server.properties
listeners property is a comma-separated list of URIs) where to listen.
The scheme (PLAINTEXT, …) is called a security protocol.
Example:
- PLAINTEXT://myhost:9092,
- SSL://:9091 ,
- CLIENT://0.0.0.0:9092,
- REPLICATION://localhost:9093
Services
Start
mkdir -p $KAFKA_HOME_DIR/logs
cd $KAFKA_HOME_DIR
nohup bin/kafka-server-start.sh config/server.properties > logs/kafka.log 2>&1 &
cd - > /dev/null
wait_for_service "kafka" 9092
where:
- wait_for_service is a netcat bash function that controls that it can make a TCP connection to the port.
More … Multibrocker
See http://kafka.apache.org/documentation.html#quickstart_multibroker