Kafka - Utility

Utility Belt

• Source
• Doc

Dub (Docker Utility Belt)

dub [-h] {template,ensure,ensure-atleast-one,wait,http-ready,path} Actions...

Ensure (Verification of an env parameter)

dub ensure [-h] name

dub ensure KAFKA_ZOOKEEPER_CONNECT
dub ensure KAFKA_ADVERTISED_LISTENERS
dub ensure KAFKA_SSL_KEYSTORE_FILENAME
dub ensure KAFKA_SSL_KEY_CREDENTIALS

Path

dub path [-h] path {writable,readable,executable,exists}

dub path /etc/kafka/ writable
dub path "$KAFKA_SSL_KEYSTORE_LOCATION" exists

Template

conversion from the env variable to a properties files through templating

dub template "/etc/confluent/docker/${COMPONENT}.properties.template" "/etc/${COMPONENT}/${COMPONENT}.properties"
dub template "/etc/confluent/docker/log4j.properties.template" "/etc/${COMPONENT}/log4j.properties"
dub template "/etc/confluent/docker/tools-log4j.properties.template" "/etc/${COMPONENT}/tools-log4j.properties"

where:

• kafka.properties.template
• log4j.properties.template
• tools-log4j.properties.template

Example: COMPONENT.properties.template. kafka.properties.template

{% set excluded_props = ['KAFKA_VERSION',
                         'KAFKA_HEAP_OPTS'
                         'KAFKA_LOG4J_OPTS',
                         'KAFKA_OPTS',
                         'KAFKA_JMX_OPTS',
                         'KAFKA_JVM_PERFORMANCE_OPTS',
                         'KAFKA_GC_LOG_OPTS',
                         'KAFKA_LOG4J_ROOT_LOGLEVEL',
                         'KAFKA_LOG4J_LOGGERS',
                         'KAFKA_TOOLS_LOG4J_LOGLEVEL']
-%}
{% set kafka_props = env_to_props('KAFKA_', '', exclude=excluded_props) -%}
{% for name, value in kafka_props.iteritems() -%}
{{name}}={{value}}
{% endfor -%}

{% set confluent_support_props = env_to_props('CONFLUENT_SUPPORT_', 'confluent.support.') -%}
{% for name, value in confluent_support_props.iteritems() -%}
{{name}}={{value}}
{% endfor -%}

becomes for kafka /etc/COMPONENT/COMPONENT.properties

broker.id=1
advertised.listeners=PLAINTEXT://localhost:29092
offsets.topic.replication.factor=1
zookeeper.connect=localhost:32181
log.dirs=/var/lib/kafka/data
listeners=PLAINTEXT://0.0.0.0:29092

Cub (Confluent Platform Utility Belt)

cub [-h] {zk-ready,kafka-ready,sr-ready,kr-ready,listeners,ensure-topic} Actions...

Example:

KAFKA_LISTENERS=$(cub listeners "$KAFKA_ADVERTISED_LISTENERS")

• Check if Zookeeper is healthy

cub zk-ready "$KAFKA_ZOOKEEPER_CONNECT" "${KAFKA_CUB_ZK_TIMEOUT:-40}"

• Check if Kafka is ready

cub kafka-ready \
    "${CONNECT_CUB_KAFKA_MIN_BROKERS:-1}" \
    "${CONNECT_CUB_KAFKA_TIMEOUT:-40}" \
    -b "$CONNECT_BOOTSTRAP_SERVERS" \
    --config /etc/"${COMPONENT}"/kafka-connect.properties