Data pipeline

Register schemas

Enterprise

This feature is only available as an add-on to the Enterprise plan.

In the Confluent ecosystem, schemas are associated with data via subjects, which are typically derived from a Kafka topic's name. For example, the default strategy, TopicNameStrategy, suffixes the topic name with -value.

You create subjects within the BSR's Confluent Schema Registry by annotating messages with a custom option that identifies the instance and subject name.

All references to buf.example.com below should be replaced with the hostname for your private BSR server.

  1. Add a dependency on the bufbuild/confluent managed module to your Buf module's buf.yaml file.

    version: v1
    name: buf.example.com/demo/analytics
    deps:
      - buf.example.com/bufbuild/confluent
    
  2. Run buf mod update to verify and lock the dependency.

    $ buf mod update
    
  3. Add one or more buf.confluent.v1.subject options to a new or existing Protobuf message, including your Confluent Schema Registry instance name and the subject's name.

    syntax = "proto3";
    
    package demo.analytics;
    
    import "buf/confluent/v1/extensions.proto";
    import "google/protobuf/timestamp.proto";
    
    message EmailUpdated {
      google.protobuf.Timestamp updated_at = 1;
      fixed64 user_id = 2;
      string previous_email = 3;
      string new_email = 4;
      bool new_email_verified = 5;
    
      option (buf.confluent.v1.subject) = {
        instance_name: "<INSTANCE_NAME>",
        name: "email-updated-value",
      }
    }
    
  4. Run buf push to update the module in the BSR. If successful, the subject is created or updated with the message as its schema.

    $ buf push
    

    You can check it in the browser using your instance URL (you must be logged into the BSR):
    https://buf.example.com/integrations/confluent/<INSTANCE_NAME>/subjects/email-updated-value/versions/latest

Backwards compatibility checks

To preserve compatibility with previous schema versions, the BSR enforces that any changes to a Confluent Schema Registry subject must not introduce a breaking change. Any backwards-incompatible changes made to a subject or its dependencies will be strictly blocked by buf push.

CSR instance lives inside the BSRCSRBSRUserCSRBSRUsertypical push flowdetect subjects& check compatibilityregister new schemas& link to subjectsalt[breaking changes][otherwise]buf pushupdate subjectsERRORERROROKOK

The Confluent Schema Registry backwards compatibility checks supersede the BSR breaking change policy enforcement settings. Other breaking changes within the module may still enter the governance review flow.

The following Buf breaking change rules are enforced on .proto files including subjects and their dependencies. The entire WIRE_JSON category is enabled, with an additional subset of rules from the FILE category:

  • WIRE_JSON
  • ENUM_NO_DELETE
  • ENUM_VALUE_NO_DELETE
  • EXTENSION_MESSAGE_NO_DELETE
  • FILE_NO_DELETE
  • RPC_NO_DELETE
  • FIELD_NO_DELETE
  • FIELD_SAME_TYPE
  • FILE_SAME_SYNTAX
  • MESSAGE_NO_DELETE
  • MESSAGE_NO_REMOVE_STANDARD_DESCRIPTOR_ACCESSOR
  • ONEOF_NO_DELETE
  • SERVICE_NO_DELETE

Deregistering schemas

To deregister a subject, remove the buf.confluent.v1.subject option from the message and re-push the module to the BSR with buf push. The subject's schema will no longer update, but will still be accessible from the BSR's Confluent Schema Registry API.