Buf Schema Registry (BSR)

Publish modules to the BSR

When you want to make a module available for other developers, you publish it so that it’s visible to your team, organization, or the wider buf community. Once you’ve published the module, developers importing its packages will be able to resolve a dependency on the module by running buf mod update. For more information, have a look at Dependency Management

Module layout

The module is a versioned unit of Protobuf files, but it's best to also incorporate a certain level of versioning in its directory and package structure.

Suppose that you are implementing the buf.build/acme/pkg module, which only contains a single .proto file initially. Rather than placing this file at the root of the module (adjacent to the buf.yaml and buf.lock files), this file should still be nested within a directory and defined with a package that attempts to make it unique across other module dependencies.

BadGood
proto/
├── buf.lock
├── buf.yaml
└── pkg.proto
proto/
├── acme
│   └── pkg
│       └── v1
│           └── pkg.proto
├── buf.lock
└── buf.yaml

For those that don't adopt this best practice, those APIs are more prone to collide with other user API definitions. For example, if a consumer needs to import Protobuf definitions from two modules, both of which define an api.proto, then the resulting module doesn't compile. In other words, it's impossible for the compiler to distinguish between what api.proto you are referring to if there are multiple.

The module layout described here is included in the MINIMAL lint category.

Publishing steps

Use the following steps to publish a module from your local machine using the buf CLI.

  1. Open a command prompt and change to your module’s root directory in the local repository.

  2. Run buf mod update, to fetch the latest digests for the specified references in the config file, and write them and their transitive dependencies to the buf.lock file.

    $ buf mod update
    
  3. Run buf build a final time to make sure everything is working.

    This builds the Protobuf files into a Buf image which is necessary for the module to be accepted into the registry

    $ buf build
    
  4. Create a branch for the module with a branch name.

    For the branch name, use a name that signals to users the nature of changes in this branch. For more, see Module versioning.

    $ buf push --branch <BRANCH_NAME>
    

    When the branch is pushed, it becomes available on the BSR, along with its documentation. Take this opportunity to validate your changes are correct. You can depend on a branch of a module using its reference in the form <module-name>:<branch-name>. For more information on branches, see Publishing branches

  5. Once satisfied, you can now push your module to the origin repository. If successful, buf will output the BSR commit hash.

    $ buf push
    
    Output
    5173e5cfeb904508839378050d95e1de

As an example, Go developers interested in your module import a package from it and run the go get command just as they would with any other module. They can run the go get command for latest versions or they can specify a particular version, as in the following example:

$ go get buf.build/gen/go/bufbuld/eliza/protocolbuffers/go@<version-or-branch-name>

To get a generated SDK that includes generated TypeScript code with protobuf-es:

$ npm install @buf/bufbuld_eliza.bufbuild_es@<version-or-branch-name>

Publishing branches

The Buf Schema Registry (BSR) stores and manages Protobuf files as versioned modules so that individuals and organizations can publish and consume their APIs without friction. However, having only a main commit history in the BSR makes it difficult for engineers to push work-in-progress modules for testing or validation in the same way that they would push commits to a git feature branch. To enable engineers to begin iterating quickly using generated code without the need to merge Protobuf schemas to main, we support BSR branches.

Branches are not included in the main commit history and work seamlessly with:

  • GitHub branches: Branches can be automatically deployed to the BSR when a change is pushed to a GitHub branch/PR by configuring a GitHub action.
  • The Buf CLI: Branches can be used locally with Buf CLI commands like buf build and buf generate.
  • Generated SDKs: Any branch can be used with generated SDKs, enabling in-progress changes to be integrated using tools like npm and go get.
illustration of the Buf Schema Registry branches workflow
Illustration of the Buf Schema Registry branches workflow

Creating branches

Users can manually push branches with a simple buf CLI command:

$ buf push --branch <BRANCH_NAME>

When the branch is pushed, it becomes available on the BSR, along with its documentation. A link at the top of the repository page will show the number of branches that were pushed to that repository and provide the ability to navigate to them:

Users can also integrate branches by adding the buf-push-action to their GitHub workflows. With this integration, users can automatically deploy branches with their Protobuf definitions to the BSR whenever they push to a GitHub branch or pull request!

To illustrate, consider this example:

name: buf-push
on: push # Apply to all pushes
jobs:
  push-module:
    # Run `git checkout`
    - uses: actions/checkout@v3
    # Install the `buf` CLI
    - uses: bufbuild/buf-setup-action@v1
    # Push the module to the BSR
    - uses: bufbuild/buf-push-action@v1
      with:
        buf_token: ${{ secrets.BUF_TOKEN }}
        # Push as a branch when not pushing to `main`
        branch: ${{ github.ref_name != 'main' }}

The branch flag indicates whether the module should be pushed as a branch. In the above example, the module will be pushed to the BSR on a branch, if the GitHub push did not occur on the main git branch. The name of the BSR branch will be the short ref name of the git branch or git tag that triggered the workflow run. For more details on buf-push-action, check out the GitHub repository and the Buf documentation.

Updating branches

Branches can be updated and built upon with multiple commits. Simply push new changes to branches, similar to creating a new branch, and the branch will be updated.

The branch reference will always resolve to the head of the branch.

Using branches with the Buf CLI

To consume the latest commit on a branch with the buf CLI, use the name of the branch as a module reference. For example, assuming a buf.gen.yaml file exists in the current directory, code can be generated for a branch with the name <branch-name> using:

$ buf generate buf.build/bufbuld/eliza:<branch-name>

The :<BRANCH_NAME> reference can also be used in other buf CLI commands including buf build, buf breaking, buf export, buf curl, or as a dependency in the buf.yaml:

version: v1
deps:
  - buf.build/bufbuld/eliza:<branch-name>
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

Please be aware that because branches can be changed and updated, they can only be used as dependencies locally. A module that has a dependency on a branch cannot be pushed to the BSR.

$ buf push
Output
Failure: failed to push module, pushing a module with a dependency pinned to a branch commit is not allowed, dependency "buf.build/connectrpc/eliza" is pinning to a branch commit "b331a4bb1be34d9ea7675ee633220777"

Using branches with generated SDKs

In addition to using branches with the Buf CLI, engineers can also consume them with generated SDKs.

To reference a branch in generated SDKs, add a @<BRANCH_NAME> reference to the go get or npm install command. The package will be generated with the latest plugin version available when referencing the name of the branch for the generated SDK.

To get a generated SDK with generated Go code using protocolbuffers/go for the branch shown above:

$ go get buf.build/gen/go/bufbuld/eliza/protocolbuffers/go@<branch-name>

To get a generated SDK that includes generated TypeScript code with protobuf-es, use the branch- prefix with the branch name:

$ npm install @buf/bufbuld_eliza.bufbuild_es@branch-<branch-name>

Publishing non-breaking API changes

Do not push backwards-incompatible changes to your module.

There are exceptions to this rule for packages in development (such as alpha and beta), but module authors should do everything they can to maintain compatibility in their module.

If, for example, the Diamond Dependency Problem manifests itself, then some users may be unable to compile their module.

The Buf CLI and the Buf Schema Registry can be configured to detect and prevent breaking changes so that API consumers can always update to the latest version of a module and never break their builds.

Publishing breaking API changes

If you absolutely must roll out a breaking change to your API, there are ways you can safely do so without breaking compatibility with your earlier module versions.

In the Module Layout example above, you'll notice the use of a versioned filepath (it contains a v1 element). In this case, the filepath reflects a versioned package that should be used in the Protobuf files in that directory (acme.pkg.v1).

This has two key benefits:

  • The Protobuf files you define don't collide with other modules so that they can always be compiled together.
  • The version element in the filepath enables you to roll out incompatible versions in the same module because they are consumed from different filepaths.

Suppose that you have a module similar to the one described in Module Layout, and you need to make a breaking change to the acme/pkg/v1/pkg.proto definitions. Rather than committing a breaking change to the same file, you can create a new file in a separately versioned filepath, such as acme/pkg/v2/pkg.proto.

What that looks like:

proto/
├── acme
│   └── pkg
│       ├── v1
│       │   └── pkg.proto
│       └── v2
│           └── pkg.proto
├── buf.lock
└── buf.yaml

In this case, acme/pkg/v2/pkg.proto is incompatible with acme/pkg/v1/pkg.proto (the Object.id field was changed):

acme/pkg/v1/pkg.proto
syntax = "proto3";

package acme.pkg.v1;

// Object is a generic object that uses
// an int32 for its identifier.
message Object {
  int32 id = 1;
}
acme/pkg/v2/pkg.proto
syntax = "proto3";

package acme.pkg.v2;

// Object is a generic object that uses
// a string for its identifier.
message Object {
  string id = 1;
}

Fortunately, with this structure, the module author can safely push their latest changes to the module and all of their consumers can continue to compile their modules.

The package version recommendation described here is enforceable in the PACKAGE_VERSION_SUFFIX lint rule which is part of the default ruleset we recommend.

Publishing commits

Every push of new content to a repository is associated with a commit that identifies that change in the schema.

The commit name is a randomly generated, fixed-size [hexadecimal] string that's visible in the BSR's UI. Note that the commit name is not a hash of the commit's content.

Commits are defined and maintained by the Buf Schema Registry and cannot be user-defined in any way. The commit is created after a successful push. This means that unlike Git, the commit only exists on the BSR repository and not locally.

To create a new commit, push your module using buf

$ buf push

Publishing tags

Tags are a user-defined reference to a single commit but with a human-readable name, similar to a Git tag. It is useful for identifying commonly referenced commits—like a release. A commit can have many tags referencing it but a tag can only reference a single commit.

$ buf push --tag <tag name>

We recommend tagging BSR commits with version control references, as a way to track corresponding revisions.

$ buf push --tag "$(git rev-parse HEAD)"