Kafka - Installation Standalone / Open Source (Single Broker)

Kafka Commit Log Messaging Process

Kafka - Installation Standalone / Open Source (Single Broker)

About

This page shows you how to install kafka from the open source package with a single broker (a single node)

Prerequisites

Kafka is working with zookeeper to store its data. A zookeeper server must be running before starting Kafka. See Zookeeper - Installation

You can see the Zookeeper version dependency in the kafka_home/libs/zookeeper-x.x.x.jar where x.x.x is the version.

Steps

The installation directories

  • The base installation directory
KAFKA_BASE_DIR=/opt
mkdir -p ${KAFKA_BASE_DIR}
  • The home directory
KAFKA_HOME_DIR=${KAFKA_BASE_DIR}/kafka

Download and unzip

  • Version and url
KAFKA_VERSION=2.1.1
KAFKA_SCALA_VERSION=2.11
KAFKA_PACKAGE_NAME=kafka_${KAFKA_VERSION}-${KAFKA_SCALA_VERSION}
KAFKA_PACKAGE_URL=https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/${KAFKA_PACKAGE_NAME}.tgz
  • Download and unzip
KAFKA_DOWNLOAD_DIR=$HOME/.kafka/download
KAFKA_DOWNLOAD_FILE="$KAFKA_DOWNLOAD_DIR/$(basename $KAFKA_PACKAGE_URL)"
curl "$KAFKA_PACKAGE_URL" > "${KAFKA_DOWNLOAD_FILE}.tmp"
mv "${KAFKA_DOWNLOAD_FILE}.tmp" "${KAFKA_DOWNLOAD_FILE}"
tar -xf "${KAFKA_DOWNLOAD_FILE}" -C $KAFKA_DOWNLOAD_DIR

Install

mv "$KAFKA_PACKAGE_NAME" "$KAFKA_HOME_DIR" 
  • Properties change via sed
# have to use SIGTERM since nohup on appears to ignore SIGINT
# and Kafka switched to SIGINT in KAFKA-1031.
sed -i.bak 's/SIGINT/SIGTERM/g' $KAFKA_HOME_DIR/bin/kafka-server-stop.sh
# in order to simplify the wikipedia-stats example job, set topic to have just 1 partition by default
sed -i.bak 's/^num\.partitions *=.*/num.partitions=1/' $KAFKA_HOME_DIR/config/server.properties

Listener changes for a local development

Properties change in the KAFKA_HOME_DIR/config/server.properties for a local development (change via sed ) on the listeners properties.

  • listeners=PLAINTEXT://0.0.0.0:9092 to listen on all interface
sed -i "s/\#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/0.0.0.0:9092/" $KAFKA_HOME_DIR/config/server.properties 
  • advertised.listeners=PLAINTEXT://localhost:9092 to redirect the client to localhost as FQDN
sed -i "s/\#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=PLAINTEXT:\/\/localhost:9092/" $KAFKA_HOME_DIR/config/server.properties

listeners property is a comma-separated list of URIs) where to listen.

The scheme (PLAINTEXT, …) is called a security protocol.

Example:

  • PLAINTEXT://myhost:9092,
  • SSL://:9091 ,
  • CLIENT://0.0.0.0:9092,
  • REPLICATION://localhost:9093

Services

Start

mkdir -p $KAFKA_HOME_DIR/logs
cd $KAFKA_HOME_DIR
nohup bin/kafka-server-start.sh config/server.properties > logs/kafka.log 2>&1 &
cd - > /dev/null
wait_for_service "kafka" 9092

where:

  • wait_for_service is a netcat bash function that controls that it can make a TCP connection to the port.

More … Multibrocker

See http://kafka.apache.org/documentation.html#quickstart_multibroker

Documentation / Reference





Discover More
Kafka Commit Log Messaging Process
Kafka - Installation

Kafka is an open source software that has two version: one for scala one for java Version kafka_2.11-2.1.1 is the version: 2.11 for Scala 2.1.1 for Java Confluent is building its business...
Yarn Hortonworks
Zookeeper - Service / Server

How to interact with the Zookeeper service/server Start Zookeeper in its own terminal. From a zookeeper archive From a kafka archive netcat wait function where run create a new...



Share this page:
Follow us:
Task Runner