Kafka - Stream Application

Kafka Commit Log Messaging Process

About

What is a Stream? Also known as Pipe, Message Queue or Event Processing in Kafka.

The stream API

The Kafka cluster stores streams of records in categories called topics.

Management

Configuration

Application Id

Application Id (application.id): Each stream processing application must have a unique id.

This id is used in the following places to isolate resources used by the application from others:

  • As the default Kafka consumer and producer client.id prefix
  • As the Kafka consumer group.id for coordination
  • As the name of the sub-directory in the state directory (cf. state.dir the directory location for state stores)
  • As the prefix of internal Kafka topic names

Update

When an application is updated, it is recommended to change application.id unless it is safe to let the updated application re-use the existing data in internal topics and state stores. One pattern could be to embed version information within application.id, e.g., my-app-v1.0.0 vs. my-app-v1.0.2.

Concept

https://docs.confluent.io/3.0.0/streams/concepts.html#streams-concepts

  • A stream processor is a node in the processor topology that represents a single processing step.
  • A stream is an unbounded, continuously updating data set.

API

Two:

Javadoc

Jar

Jar

Group Id Artifact Id Description / why needed
org.apache.kafka kafka-streams Base library for Kafka Streams. Required.
org.apache.kafka kafka-clients Kafka client library. Contains built-in serializers/deserializers. Required.
org.apache.avro avro Apache Avro library. Optional (needed only when using Avro).
io.confluent kafka-avro-serializer Confluent’s Avro serializer/deserializer. Optional (needed only when using Avro).

Code / Demo

  • Code examples that demonstrate how to implement real-time processing applications using Kafka Streams. See readme.

Documentation / Reference





Discover More
Kafka Commit Log Messaging Process
Kafka (Event Hub)

Apache Kafka is a broker application that stores the message as a distributed commit log. The entire data storage system is just a transaction log. |data feeds Data Systems are exposing data, ...
Log Consumer
Kafka - (Consumer) Offset

The offset is the position of a consumer in a topic keyrecord Zookeeperconsumer groupStream Processing ...
Kafka Commit Log Messaging Process
Kafka - (Record|Message)

in Kafka. The Kafka cluster stores streams of records in categories called topics. Each record consists of: a key, a value, and a timestamp. See built-in timestamp org/apache/kafka/connect/data/Structorg.apache.kafka.connect.data.Struct...
Kafka Commit Log Messaging Process
Kafka - API

Kafka has four core APIs: The Producer API allows an application to publish a stream of records to one or more Kafka topics. The Consumer API allows an application to subscribe to one or more topics...
Log Consumer
Kafka - Consumer

A consumer. A sink connector is a consumer. kafka-console-consumer is a command line to read data from a Kafka topic and write it to standard output. The configs can be overridden by...
Kafka Commit Log Messaging Process
Kafka - Partition

in Kafka Each partition is an: ordered, immutable sequence of records that is continually appended to—a structured commit log. The records in the partitions are each assigned a sequential...
Kafka Commit Log Messaging Process
Kafka - Stream Word Count demo

Stream Word count demo adapted from the stream quickstart. Install docker: Map the hostname zookeeper and broker to your docker host...
Log Anatomy
Kafka - Topic

The Kafka cluster stores streams of records in categories called topics. A topic is also known as: a category or feed name. A topic can have zero, one, or many consumers that subscribe to the data...
Converter Basics
Kafka Connect - Transform (Single Message Transform - SMT)

Transform is a concept of connect to apply simple transformation. A transform is a simple function that accepts one record as input and outputs a modified record. Transform are chained. In case of...



Share this page:
Follow us:
Task Runner