Kafka - Consumer

console utility

• kafka-console-consumer is a command line to read data from a Kafka topic and write it to standard output.
• Kafka - kafka-avro-console-consumer utility

Configuration

The configs can be overridden by prefixing them with consumer.

For example:

consumer.max.partition.fetch.bytes=10485760

http://kafka.apache.org/documentation.html#newconsumerconfigs

Offset

The only metadata retained on a per-consumer basis is the offset or position of that consumer in the log.

This offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads records, but, in fact, since the position is controlled by the consumer it can consume records in any order it likes.

For example a consumer can reset to an older offset to reprocess data from the past or skip ahead to the most recent record and start consuming from “now”.

Group

The consumer group is used for coordination between consumers. See Kafka - Consumer Group

Message Read Status

When all in-sync replicas have acknowledged the write, then the message is considered committed, which makes it available for reading. This ensures that messages cannot be lost by a broker failure after they have already been read.