Skip to content

Integrate with Kafka – Overview

This feature is only available on the Enterprise plan.

The Buf Schema Registry Kafka integration is a Protobuf-first implementation of the Confluent Schema Registry (CSR) APIs within the Buf Schema Registry (BSR). It's designed to strengthen your Kafka pipeline data quality where it helps the most—at build time, and at the source of truth for your schemas. With Buf's Data Pipeline Add-on, you can take advantage of powerful stream processing systems like kSQL and Kafka Connect without the risk of runtime schema mismatch errors.

  • Creating your CSR instance is a 1-step process—we've implemented the CSR API so you don't have to, and instances can be easily managed by your BSR admins.
  • Our CSR maps Confluent subjects to messages in your schema with straightforward annotations—no migration or conversion to other schema formats required. It takes all the power of the open-source CSR and makes it easy to use within your existing BSR workflow.
  • Buf's breaking change detection validates that schema changes are backward-compatible in IDEs, GitHub pull requests, and as they're published to the BSR.
  • The CSR integration works with most Kafka producers and consumers, downstream systems like kSQL and Kafka Connect, and management tools like AKHQ, making all the capabilities of the Confluent ecosystem available to your organization.

Getting started

Taking advantage of Buf's CSR implementation has three overall steps:

  1. Once you've upgraded to the add-on and your Buf representative has enabled it, go to the BSR and create your CSR instance.

  2. Associate your schemas with Confluent subjects by integrating the bufbuild/confluent managed module and annotating your Protobuf messages (see Register schemas for detailed steps).

    syntax = "proto3"

package demo.analytics;

import "buf/confluent/v1/extensions.proto";
import "google/protobuf/timestamp.proto";

message EmailUpdated {
  google.protobuf.Timestamp updated_at = 1;
  fixed64 user_id = 2;
  string previous_email = 3;
  string new_email = 4;
  bool new_email_verified = 5;

  option (buf.confluent.v1.subject) = {
    instance_name: "default",
    name: "email-updated-value",
  };
}

  3. Push the annotated .proto files up to the BSR. Buf's breaking change detection validates the annotated schemas and blocks the push if they're not backward compatible.

Integrating with Kafka clients

Once the push that includes the subject mapping is successful, the Confluent integration automatically creates a subject associated with the message on your CSR instance. For example, given the example .proto file above, the Confluent integration automatically creates a subject named email-updated-value associated with the demo.analytics.EmailUpdated message on the default CSR instance.

Because subjects are defined in your schemas, you're able to view, code review, and manage them the way you would any other source code. In the BSR, you could view the subject above at:

https://buf.example.com/integrations/confluent/default/subjects/email-updated-value/versions/latest

After you create the subject, you use the CSR URL and a Buf token to configure Confluent-aware Kafka producers and consumers to serialize and deserialize the topics defined in your Protobuf schemas. See Integrating with Kafka clients for examples.

Breaking change detection and compatibility guarantees

The CSR integration works with Buf's existing breaking change detection and server-wide policy check to ensure backwards compatibility and identify problems early in your development cycle:

  • Developers can enable one of Buf's editor plugins to enforce formatting, linting, and breaking change rules locally and test as they're building schemas. Buf's FILE setting for breaking change detection detects all breakages relevant to Kafka.

  • Your organization can hook into Buf's GitHub Actions to enforce breaking change detection before schemas containing subjects are merged into the default label and registered in the BSR. Again, Buf's FILE-level breaking change detection guarantees that changes are safe for Kafka.

  • Finally, with the Data Pipeline Add-on enabled, on buf push the BSR enforces that any changes to a CSR subject won't introduce breaking changes, ensuring that all schemas are valid and compatible at build time. This is a final safety check that's applied regardless of other settings earlier in the development process, and uses a special set of rules to address CSR concerns.

Next steps