Kafka - Schema

Kafka Commit Log Messaging Process

About

You can use an schema, for example, to:

Syntax

An Avro schema defines the data structure in a JSON format.

{"namespace": "example.avro",
 "type": "record",
 "name": "user",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": "int"}
 ]
}

where:

Management

Schema management is done through the schema registry service

List all schemas (subject ?)

Example made with sqlite getting started

curl -X GET http://localhost:8081/subjects
["test-sqlite-jdbc-accounts-value"]

List all schema versions

List all schema versions registered under a subject. Example made with sqlite getting started

curl -X GET http://localhost:8081/subjects/subjectName/versions

# Example
curl -X GET http://localhost:8081/subjects/test-sqlite-jdbc-accounts-value/versions
[1,2]

Fetch a schema

Fetch version n of the schema registered under the subject. Example made with sqlite getting started

curl -s -X GET http://localhost:8081/subjects/subjectName/versions/n | jq .

# Example
curl -s -X GET http://localhost:8081/subjects/test-sqlite-jdbc-accounts-value/versions/1 | jq .
{                                                                                                                                                                                                          
  "subject": "test-sqlite-jdbc-accounts-value",                                                                                                                                                            
  "version": 1,                                                                                                                                                                                            
  "id": 1,                                                                                                                                                                                                 
  "schema": "{\"type\":\"record\",\"name\":\"accounts\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null}],\"connect.name\":\"accounts\"}"                                                                                                                                                                                                          
}

The schema property properly indented is:

{
	"type ": "record",
	"name":"accounts",
	"fields": [{
			"name":"id",
			"type":"long"
		}, {
			"name":"name",
			"type": ["null","string"],
			"default": null
		}
	],
	"connect.name":"accounts"
}

Delete

# Deletes all schema versions registered under the subject "Kafka-value"
$ curl -X DELETE http://localhost:8081/subjects/Kafka-value
  [1]

# Deletes version 1 of the schema registered under subject "Kafka-value"
$ curl -X DELETE http://localhost:8081/subjects/Kafka-value/versions/1
  1

# Deletes the most recently registered schema under subject "Kafka-value"
$ curl -X DELETE http://localhost:8081/subjects/Kafka-value/versions/latest
  1

Documentation / Reference





Discover More
Kafka Commit Log Messaging Process
Kafka - (Record|Message)

in Kafka. The Kafka cluster stores streams of records in categories called topics. Each record consists of: a key, a value, and a timestamp. See built-in timestamp org/apache/kafka/connect/data/Structorg.apache.kafka.connect.data.Struct...
Kafka Commit Log Messaging Process
Kafka - Avro Converter

in Kafka. The Avro converter is normally used with the Schema Registry in order to properly lookup the schema for the Avro data. ...
Kafka Commit Log Messaging Process
Kafka - JSON Converter

JSON converter The value.converter.schemas.enable and/or key.converter.schemas.enable must be set to false to have the JSON contain only they value and key, respectively, without the schemas.
Kafka Commit Log Messaging Process
Kafka - Oracle

The connector for oracle There is actually two given by Oracle: the OGG (Oracle GoldenGate ) Kafka Handler (Supported), or an open source Kafka Connect handler And others: Xstream...
Kafka Commit Log Messaging Process
Kafka - Schema Registry

schema registry Schema registry is recommended if you plan to use Avro for a data format because it can help you with: serialization and schema evolution. with schema knowledge the Avro...
Kafka Commit Log Messaging Process
Kafka - Tutorials

A list of tutorials that I made on windows. They were my path to learn Kafka confluent In this tutorial: you start all services in a docker container you write and read data into a topic...
Kafka Commit Log Messaging Process
Kafka - kafka-avro-console-consumer utility

The kafka-avro-console-consumer is a the kafka-console-consumer with a avro formatter (io.confluent.kafka.formatter.AvroMessageFormatter) This console uses the Avro converter with the Schema Registry...
Kafka Commit Log Messaging Process
Kafka - kafka-avro-console-producer utility

The kafka-avro-console-producer is a producer command line to read data from standard input and write it to a Kafka topic in an avro format. This console uses the Avro converter with the Schema Registry...



Share this page:
Follow us:
Task Runner