Schema Evolution in Kafka: How to Design Resilient Event Contracts in Go

This content originally appeared on DEV Community and was authored by Vlad Pistun

In distributed systems, change is inevitable – but breaking things doesn’t have to be. As your system evolves, so will the data exchanged between services. In Kafka-based architectures, that evolution must be handled carefully. Fail to do so, and you’ll end up with incompatible consumers, corrupted data, and replay nightmares.

This article dives into schema evolution in Kafka, with a sharp focus on using Protobuf and Go to build resilient contracts. We’ll cover versioning strategies, evolution-safe patterns, common pitfalls, and practical examples using kafka-go and a schema registry (Buf or Confluent).

Why Schema Evolution Matters

When your services communicate through Kafka topics, they rely on a shared contract – the event schema. But these schemas aren’t static. You’ll inevitably need to:

• Add new fields (e.g., tax_code to an Order event)
• Deprecate fields (e.g., replacing total_amount with tax_amonut + taxable_amount)
• Change types (e.g., int → decimal)

Without a disciplined approach to schema evolution, these changes break consumers, especially if they’re deployed out-of-sync or written in different languages.

Breaking change example:
You remove a field that a downstream Go service still expects. On replay, it panics, or worse, silently processes corrupted data.

Kafka Doesn’t Understand Your Schema

Kafka stores bytes – not meaning. That means schema evolution isn’t Kafka’s problem to solve. It’s yours.

To manage this, most production-grade systems use a Schema Registry:

• Confluent Schema Registry (for Avro, Protobuf, JSON Schema)
• Buf Schema Registry

These registries:

• Store schema versions
• Check for compatibility (forward, backward, full)
• Help consumers safely deserialize bytes

Protobuf: The Go-Friendly Choice

For Go services, Protobuf offers strong typing, excellent performance, and good evolution support.

Why not JSON or Avro?

• JSON: No schema enforcement. Easy to break without noticing.
• Avro: Schema evolution support is solid, but Go support is weak and verbose.

Why Protobuf Works Well

• Fields can be marked optional
• Defaults are implicit
• Fields have tags (field numbers), so renaming is safe
• Decoding skips unknown fields by default

Designing for Evolution: Best Practices

1. Add Fields, Don’t Remove
Additive changes are safest. Make new fields optional, and give them safe defaults on the consumer side.

message OrderV2 {
  string id = 1;
  int32 amount = 2;
  optional int32 tax = 3; // added in v2
}

In Go:

type OrderV2 struct {
  Id     string
  Amount int32
  Tax    *int32 // optional, nil if not present
}

2. Don’t Reuse Field Numbers
In Protobuf, field numbers are sacred. Changing the type of an existing field or reusing an old number for a new field leads to unpredictable behavior.

3. Default Tolerant Consumers
Consumers should handle missing fields gracefully and never rely on a field always being present.

Forward vs Backward Compatibility

Topic vs Schema Versioning

There are two major strategies:

1. Versioned Schema, Same Topic

• Topic remains orders
• Schemas evolve (v1, v2, v3…)
• Consumers must be aware of schema versions or use registry-based decoding

2. Versioned Topics (orders.v1, orders.v2)

• Clear boundary, simpler consumers
• Harder to replay across versions
• Сan lead to topic sprawl

Recommended: Use versioned schemas + a schema registry, unless you have a good reason to version topics.

Conclusion

Schema evolution in Kafka is a challenge – but not an unsolvable one. With Protobuf, a proper schema registry, and evolution-safe Go code, you can build contracts that withstand change and avoid production surprises.

Design schemas like APIs. Make them clear, compatible, and built to last.

Furthermore: Exploring Buf Stream

Buf recently introduced Buf Stream, a high-performance streaming schema transport designed to simplify Protobuf communication across services. While not yet widely adopted in Kafka ecosystems, it shows promise for future use cases involving real-time schema-bound messaging – especially where tight Protobuf integration and compatibility checks are critical.

At the time of writing, we haven’t tested Buf Stream in production or in Kafka contexts, but it’s worth keeping an eye on as the ecosystem evolves.

This content originally appeared on DEV Community and was authored by Vlad Pistun