Kafka - Stream Application

Kafka Commit Log Messaging Process

About

(Stream|Pipe|Message Queue|Event Processing) in Kafka.

The stream API

The Kafka cluster stores streams of records in categories called topics.

Management

Configuration

Application Id

Application Id (application.id): Each stream processing application must have a unique id.

This id is used in the following places to isolate resources used by the application from others:

  • As the default Kafka consumer and producer client.id prefix
  • As the Kafka consumer group.id for coordination
  • As the name of the sub-directory in the state directory (cf. state.dir the directory location for state stores)
  • As the prefix of internal Kafka topic names

Update

When an application is updated, it is recommended to change application.id unless it is safe to let the updated application re-use the existing data in internal topics and state stores. One pattern could be to embed version information within application.id, e.g., my-app-v1.0.0 vs. my-app-v1.0.2.

Concept

https://docs.confluent.io/3.0.0/streams/concepts.html#streams-concepts

  • A stream processor is a node in the processor topology that represents a single processing step.
  • A stream is an unbounded, continuously updating data set.

API

Two:

Javadoc

Jar

Jar

Group Id Artifact Id Description / why needed
org.apache.kafka kafka-streams Base library for Kafka Streams. Required.
org.apache.kafka kafka-clients Kafka client library. Contains built-in serializers/deserializers. Required.
org.apache.avro avro Apache Avro library. Optional (needed only when using Avro).
io.confluent kafka-avro-serializer Confluent’s Avro serializer/deserializer. Optional (needed only when using Avro).

Code / Demo

  • Code examples that demonstrate how to implement real-time processing applications using Kafka Streams. See readme.
  • Demo

Documentation / Reference




Discover More
Kafka Commit Log Messaging Process
Kafka (Event Hub)

Apache Kafka is a broker application that stores the message as a distributed commit log. The entire data storage system is just a transaction log. |data feeds Data Systems are exposing data, ...
Log Consumer
Kafka - (Consumer) Offset

The offset is the position of a consumer in a topic keyrecord Zookeeperconsumer groupStream Processing ...
Kafka Commit Log Messaging Process
Kafka - (Record|Message)

in Kafka. The Kafka cluster stores streams of records in categories called topics. Each record consists of: a key, a value, and a timestamp. See built-in timestamp org/apache/kafka/connect/data/Structorg.apache.kafka.connect.data.Struct...
Kafka Commit Log Messaging Process
Kafka - API

Kafka has four core APIs: The Producer API allows an application to publish a stream of records to one or more Kafka topics. The Consumer API allows an application to subscribe to one or more topics...
Log Consumer
Kafka - Consumer

A consumer. A sink connector is a consumer. kafka-console-consumer is a command line to read data from a Kafka topic and write it to standard output. The configs can be overridden by...
Kafka Commit Log Messaging Process
Kafka - Partition

in Kafka Each partition is an: ordered, immutable sequence of records that is continually appended to—a structured commit log. The records in the partitions are each assigned a sequential...
Kafka Commit Log Messaging Process
Kafka - Stream Word Count demo

Stream Word count demo adapted from the stream quickstart. Install docker: Map the hostname zookeeper and broker to your docker host...
Log Anatomy
Kafka - Topic

The Kafka cluster stores streams of records in categories called topics. A topic is also known as: a category or feed name. A topic can have zero, one, or many consumers that subscribe to the data...
Converter Basics
Kafka Connect - Transform (Single Message Transform - SMT)

Transform is a concept of connect to apply simple transformation. A transform is a simple function that accepts one record as input and outputs a modified record. Transform are chained. In case of...



Share this page:
Follow us:
Task Runner