Skip to content

Integrate with Kafka – Overview

This feature is only available on the Enterprise plan.

The Buf Schema Registry Confluent Schema Registry integration is a Protobuf-first implementation of the Confluent Schema Registry (CSR) APIs within the Buf Schema Registry (BSR). It's designed to strengthen your streaming data quality where it helps the most—at build time, and at the source of truth for your schemas. Because the BSR implements the entirety of the CSR API, you can take advantage of the Confluent ecosystem including stream processing systems like kSQL and Kafka Connect without the risk of runtime errors.

  • Creating a CSR instance is a 1-step process—we've implemented the CSR API so you don't have to, and instances can be easily managed by your BSR admins.
  • The BSR maps Confluent subjects to messages in your schema with straightforward annotations—no migration or conversion to other schema formats required. It brings the breadth and power of the Confluent ecosystem into your existing BSR workflow.
  • Buf's breaking change policy checks validate that schema changes are backward compatible in IDEs, GitHub pull requests, and as they're published to the BSR, ensuring that only safe and approved changes flow downstream.
  • The CSR integration works with any Confluent-compatible Kafka client, downstream systems like kSQL and Kafka Connect, and management tools like AKHQ, making all the capabilities of the Confluent ecosystem available to your organization.

Getting started

Taking advantage of Buf's CSR implementation has three overall steps:

  1. Go to the BSR and create your CSR instance.

  2. Associate your schemas with Confluent subjects by integrating the bufbuild/confluent managed module and annotating your Protobuf messages (see Register schemas for detailed steps).

    syntax = "proto3"

package demo.analytics;

import "buf/confluent/v1/extensions.proto";
import "google/protobuf/timestamp.proto";

message EmailUpdated {
  google.protobuf.Timestamp updated_at = 1;
  fixed64 user_id = 2;
  string previous_email = 3;
  string new_email = 4;
  bool new_email_verified = 5;

  option (buf.confluent.v1.subject) = {
    instance_name: "default",
    name: "email-updated-value",
  };
}

  3. Push the annotated .proto files up to the BSR. Buf's breaking change detection validates the annotated schemas and blocks the push if they're not backward compatible.

Integrating with Kafka clients

Once the push that includes the subject mapping is successful, the Confluent integration automatically creates a subject associated with the message on your CSR instance. For example, given the example .proto file above, the Confluent integration automatically creates a subject named email-updated-value associated with the demo.analytics.EmailUpdated message on the default CSR instance.

Because subjects are defined in your schemas, you're able to view, code review, and manage them the way you do any other source code. In the BSR, you could view the subject above at:

https://buf.example.com/integrations/confluent/default/subjects/email-updated-value/versions/latest

After you create the subject, you use the CSR URL and a Buf token to configure Confluent-aware Kafka producers and consumers to serialize and deserialize the topics defined in your Protobuf schemas. See Integrating with Kafka clients for examples.

Breaking change detection and compatibility guarantees

The CSR integration works with Buf's existing breaking change detection and instance-wide policy check to ensure backwards compatibility and identify problems early in your development cycle:

  • Developers can enable one of Buf's editor plugins to enforce formatting, linting, and breaking change rules locally and test as they're building schemas. Buf's FILE setting for breaking change detection detects all breakages relevant to Kafka.

  • Your organization can hook into Buf's GitHub Actions to enforce breaking change detection before schemas containing subjects are merged into the default label and registered in the BSR. Again, Buf's FILE-level breaking change detection guarantees that changes are safe for Kafka.

  • Finally, with the CSR integration enabled, on buf push the BSR enforces that any changes to a CSR subject won't introduce breaking changes, ensuring that all schemas are valid and compatible at build time. This is a final safety check that's applied regardless of other settings earlier in the development process, and uses a special set of rules to address CSR concerns.

Next steps