Kafka - (Consumer) Offset

Kafka Commit Log Messaging Process

About

The offset is the position of a consumer in a topic

An offset is not the key but an automatic record position id

For versions less than 0.9 Apache Zookeeper was used for managing the offsets of the consumer group. Offset management is the mechanism, which tracks the number of records that have been consumed from a partition of a topic for a particular consumer group. Kafka 0.10 came out with out of the box support for Stream Processing.

Log Consumer

Management

Reset tool

https://kafka.apache.org/25/documentation/streams/developer-guide/app-reset-tool.html

Connect

In a distributed mode, connect registers the offset of each connector in a special topic called connect-offsets

with Kafka - kafka-console-consumer

kafka-console-consumer \
--bootstrap-server localhost:9092 \
--from-beginning \
--property print.key=true \
--topic connect-offsets

Example output that you get following the file-source demo

["file-source",{"filename":"test.txt"}] {"position":75}
["file-source",{"filename":"test.txt"}] {"position":150}
["file-source",{"filename":"test.txt"}] {"position":225}

Check

How to check the number of messages read and written, as well as the lag for each consumer in a specific consumer group.

ConsumerGroupCommand

kafka-consumer-groups --bootstrap-server broker:9092 --describe --group console-consumer-36650
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

Consumer group 'console-consumer-36650' has no active members.

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
test-sqlite-jdbc-accounts      0          2               8               6          -                                                 -                              -

kafka-consumer-offset-checker

ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0, use the ConsumerGroupCommand below instead. (kafka.tools.ConsumerOffsetChecker$)

kafka-consumer-offset-checker --group consumer-group --topic test-sqlite-jdbc-accounts --zookeeper zookeeper:2181

GetOffsetShell

An interactive shell for getting consumer offsets.

kafka-run-class kafka.tools.GetOffsetShell
Option                                 Description                                 
------                                 -----------                                 
--broker-list <String: hostname:       REQUIRED: The list of hostname and          
  port,...,hostname:port>                port of the server to connect to.         
--max-wait-ms <Integer: ms>            The max amount of time each fetch           
                                         request waits. (default: 1000)            
--offsets <Integer: count>             number of offsets returned (default: 1)     
--partitions <String: partition ids>   comma separated list of partition ids.      
                                         If not specified, it will find            
                                         offsets for all partitions (default:      
                                         )                                         
--time <Long: timestamp/-1(latest)/-2  timestamp of the offsets before that        
  (earliest)>                            (default: -1)                             
--topic <String: topic>                REQUIRED: The topic to get offset from.     

Example:

kafka-run-class kafka.tools.GetOffsetShell --broker-list "broker:9092" --topic mytopic
mytopic:0:201

ExportZkOffsets

Retrieves the offsets of broker partitions in zookeeper and prints to an output file

ExportZkOffsets is deprecated

kafka-run-class.sh kafka.tools.ExportZkOffsets
WARN WARNING: ExportZkOffsets is deprecated and will be dropped in a future release following 0.11.0.0. (kafka.tools.ExportZkOffsets$)
Export consumer offsets to an output file.

Option                  Description
------                  -----------
--group <String>        Consumer group.
--help                  Print this message.
--output-file <String>  Output file
--zkconnect <String>    ZooKeeper connect string. (default: localhost:2181)





Discover More
Kafka Commit Log Messaging Process
Kafka - (Record|Message)

in Kafka. The Kafka cluster stores streams of records in categories called topics. Each record consists of: a key, a value, and a timestamp. See built-in timestamp org/apache/kafka/connect/data/Structorg.apache.kafka.connect.data.Struct...
Log Consumer
Kafka - Consumer

A consumer. A sink connector is a consumer. kafka-console-consumer is a command line to read data from a Kafka topic and write it to standard output. The configs can be overridden by...
Kafka Commit Log Messaging Process
Kafka - Consumer Group

The consumer group is used for coordination between consumer The consumer group is given by the group.id configuration property of a consumer. A console will generate automatically one for you with...
Kafka Commit Log Messaging Process
Kafka - Ksql

Without the LIMIT keyword, the SELECT query would run indefinitely until you stop it by pressing ==== Persistent Query ==== Unlike the non-persistent query above, Results from this query are written...
Kafka Commit Log Messaging Process
Kafka - Partition

in Kafka Each partition is an: ordered, immutable sequence of records that is continually appended to—a structured commit log. The records in the partitions are each assigned a sequential...
Kafka Commit Log Messaging Process
Kafka - Zookeeper

Zookeeper in Kafka. It is one of the Kafka service. List of broker List of topics List of replicas of topics Consumer offset Leader election algorithm Clean up Zookeeper data folder...
Kafka Commit Log Messaging Process
Kafka - kafka-console-consumer

kafka-console-consumer is a consumer command line that: read data from a Kafka topic and write it to standard output (console). Example with docker Option Description Example ...
Converter Basics
Kafka Connect - Converter (Read/Write - I/O)

A converter is a connect concept. It's the code used to persist data from a Connector. Converters are decoupled from connectors themselves to allow for reuse. For example, using the same Avro...
Worker Model Basics
Kafka Connect - Distributed Worker

The runtime distributed mode of connect when running/starting a worker See where: worker.properties is the configuration file New workers will either start a new group or join an existing...
Kafka Commit Log Messaging Process
Kafka Connect - File

File connector Built-in Connector Both source and sink connectors can track offsets, so you can start and stop the process any number of times and add more data to the input file and both...



Share this page:
Follow us:
Task Runner