Try the Buf Schema Registry

This tour demonstrates the workflow and features of the Buf Schema Registry (BSR), which helps organizations manage schemas, dependencies, and governance at scale. You'll create a BSR repository, push a module of Protobuf files, and learn how it handles dependency management and generating API documentation. Finally, you'll put it all together to build an API client using SDKs that the BSR auto-generates from the module.

Prerequisites

• Install the Buf CLI if you haven't already. You need version 1.32 or higher to do the tour, so if you previously installed the Buf CLI, check the version and update if necessary:

  $ buf --version
  

• Have git and go installed and in your $PATH.

• Clone the buf-tour repo:

  $ git clone git@github.com:bufbuild/buf-tour.git
  

• Sign up for a Buf account if you haven't already. A Buf account is required to access and share BSR repositories and allows you to explore modules that are available from the community and verified publishers.

The repository contains a start directory and a finish directory. During this tour you'll work on files in the start/getting-started-with-bsr directory, and at the end they should match the files in the finish/getting-started-with-bsr directory.

1. Create an API token and sign in

Create a token

1. Sign in to the BSR. You'll be prompted with a few options, including Google, GitHub, and traditional email and password. You've successfully logged in if your username is shown in the upper right-hand corner.

2. Go to the the User settings page.

3. Click the Create New Token button.

4. Select an expiration time. You can optionally add a note for yourself to distinguish this token from others (we recommend that you name this CLI, Development, or something else along those lines).

5. Click Create.

6. Copy the token to your clipboard. You'll use it for the rest of the tour.

Sign in from the command line

Use the API token you created to sign in:

$ buf registry login
Output
Log in with your Buf Schema Registry username. If you don't have a username, create one at https://buf.build.

Username: <USERNAME>
Token: <YOUR TOKEN>

2. Push a module

Now that you've authenticated with the BSR, you can create a repository and push the provided module that defines the PetStoreService API.

Modules are the core primitive of Buf and the BSR. A module is a collection of Protobuf files that are configured, built, and versioned as a logical unit. The buf.yaml file in the tour repo initializes the module, and the pet/v1/pet.proto is the API schema.

Create a BSR repository

A module is stored in the BSR as a repository. Repositories store all versions of a module, where each version is identified by a commit and (optionally) a label. Though roughly analogous to a Git repository, a BSR repository is only a remote storage location—there is no concept of a repository "clone". In other words, BSR repositories do not exist in multiple locations.

To create a new repository:

1. Go to the home page of the BSR.
2. Select your username in the top right corner.
3. Select Repositories from the dropdown.
4. Click the Create repository button.
5. Name the repository petapis. Keep the repository public.

You should see an empty repository named petapis. Next up, you'll push a module. For the purposes of this tutorial, please skip the suggested commands on the screen and continue to the next step.

Configure the module name

Next, you need to configure your local module (provided in the sample code) so that its history will be stored in the repository you just created. This connection is made via the name field in the module's buf.yaml configuration file, which must include the full path to the repository (BSR server, organization, and repository name).

The buf.yaml file can list multiple modules, each of which corresponds to a single repository on the BSR. This example only has one, which is contained in the proto subdirectory.

Add a name field to your buf.yaml file that matches the repository you just created:

version: v2
modules:
  - path: proto
+   name: buf.build/<USERNAME>/petapis
lint:
  use:
    - DEFAULT
  ignore:
    - proto/google/type/datetime.proto
breaking:
  use:
    - FILE

Push the module

Now push the module to the buf.build/<USERNAME>/petapis repository:

$ buf push
Output
buf.build/<USERNAME>/petapis:4b28576bedf545cba5fa30ba741d1c6d

Pushing the module creates a commit in the BSR, which returns a commit ID to the CLI. Your value will differ.

Behind the scenes, the Buf CLI recognizes the name in your buf.yaml and pushes the module to the buf.build/<USERNAME>/petapis repository. If successful, the generated commit identifies this version of your module.

3. View and edit documentation

When you push a module to the BSR, it automatically generates documentation for all messages, fields, enums, and services defined in your .proto files. You can browse the generated documentation for your module by going to the Docs page in your repository at https://buf.build/:username/petapis/docs.

Add module-level documentation

The page you see above serves as the primary entry point for your module's documentation. But as you can see from the default content, you currently don't have any module-level documentation.

You can can add it by creating a README.md file in the root directory of your module (proto in this example), and pushing the module to the BSR. The README.md file is analogous to a GitHub repository's README.md file and supports all CommonMark syntax.

Create the file:

$ touch README.md

Then copy/paste this Markdown content into it to create an introductory note:

## PetAPIs

This module contains all the APIs required to interact with the `PetStoreService`.

Your proto directory should now look like this:

proto
├── google
│   └── type
│       └── datetime.proto
├── pet
│   └── v1
│       └── pet.proto
└── README.md

Push the module again, and if you refresh the documentation page you visited above, you should see the changes you just introduced.

View package documentation

As you can see from the module documentation page, both the pet.v1 and google.type packages are available as links. Click on the pet.v1 link to navigate to its package documentation. From here, you can click through each of the Protobuf type definitions and see all the comments associated with each type. In fact, if you click on the google.type.DateTime message referenced in the Pet message, you'll be brought to the google.type.v1 package documentation for the same commit.

4. Add a dependency

Without the BSR, you can only depend on other Protobuf APIs by manually fetching the .proto files you need. For example, if you want to use googleapis, you need to clone the right Git repository and copy the .proto files locally to compile your own .proto files. And if googleapis has its own external dependencies, you need to fetch those as well.

This way of managing dependencies is prone to API drift. If the googleapis code evolves over time, your local copies are inconsistent with the latest version and your modules become out of date. In the example code, it turns out that this is exactly what happened: the google/type/datetime.proto file is present in your local directory and used to build your module.

Now that you're familiar with the BSR, you can simplify this entire workflow immensely.

Remove the datetime.proto file

Start by removing the google/type/datetime.proto file from your module. From within the proto directory, run this command to remove the local google dependencies:

$ rm -r google

Now remove the google/type/datetime.proto reference from your buf.yaml file:

version: v2
modules:
  - path: proto
    name: buf.build/<USERNAME>/petapis
lint:
  use:
    - DEFAULT
-  ignore:
-    - google/type/datetime.proto
breaking:
  use:
    - FILE

If you try to build the module in its current state, you get an error:

$ buf build
Output
pet/v1/pet.proto:5:8:import "google/type/datetime.proto": file does not exist

Depend on googleapis

You resolve this error by adding a dependency in your buf.yaml file's deps key. The google/type/datetime.proto file is available in the BSR-hosted buf.build/googleapis/googleapis module, so you can configure it like this:

version: v2
modules:
  - path: proto
    name: buf.build/<USERNAME>/petapis
lint:
  use:
    - DEFAULT
breaking:
  use:
    - FILE
+deps:
+  - buf.build/googleapis/googleapis

If you try to build the module again, you'll get the same warning, because even though you've specified the dependency, you don't yet have a buf.lock dependency manifest that pins the dependency to a specific version. To create one, run this command:

$ buf dep update

The buf dep update command updates all of your deps to their latest version. The generated buf.lock file should look similar to this (the commit ID and digest will vary):

# Generated by buf. DO NOT EDIT.
version: v2
deps:
  - name: buf.build/googleapis/googleapis
    commit: 4ed3bc159a8b4ac68fe253218760d035
    digest: b5:74a7798987b123218c004cf28543a2835e432ca04a69de99cd394a29dbad24d9ed38344f0b7c97ad6476039506c4eb38c2f4a8eef9cec3da2e38e4216a22d495

When you try to build the module again, it should be successful. A few things happened here, so let's break it down:

1. The Buf CLI noticed that a new dependency was added to the deps key.
2. It resolved the latest version of the buf.build/googleapis/googleapis module and wrote it to the module's buf.lock file.
3. When you ran buf build again , it downloaded the buf.build/googleapis/googleapis module to the local module cache.
4. Once all of the dependencies were available locally, it successfully built the module (because google/type/datetime.proto was in the local module cache).

In summary, you can define your dependencies once in your buf.yaml file for all modules in the workspace, and the BSR will resolve the dependencies to include the imports required to build the modules. You don't have to manually copy .proto files any more!

Push your changes

Now that you've updated your module to depend on buf.build/googleapis/googleapis instead of vendoring the google/type/datetime.proto yourself, you can push the module to the BSR:

$ buf push
Output
buf.build/<USERNAME>/petapis:e3c35074dc9a43989d9f5c546a036d8b

Navigate back to your repository on the BSR and you'll notice your packages have changed. The Google package is no longer a first-class citizen of your module—it's now a third-party dependency. You can see the external dependencies for any module by going to its Deps tab:

This is what a dependency on googleapis looks like. Note that it's pinned to a specific commit of googleapis—which for your module matches the one listed in your buf.lock file.

Update the buf.gen.yaml file and regenerate code

Your gen/ directory should look like this

gen
├── google
│   └── type
│       └── datetime.pb.go
└── pet
    └── v1
        ├── pet.pb.go
        └── petv1connect
            └── pet.connect.go

Now that you've exchanged your local copy of googleapis for the one on the BSR, you can remove the generated code also. First, remove the gen/ directory:

$ cd ..
$ rm -r gen

Then update the buf.gen.yaml to exclude overriding any Go import statements related to googleapis.

version: v2
managed:
  enabled: true
+ disable:
+   - module: buf.build/googleapis/googleapis
  override:
    - file_option: go_package_prefix
      value: github.com/bufbuild/buf-tour/gen
plugins:
  - remote: buf.build/protocolbuffers/go
    out: gen
    opt: paths=source_relative
  - remote: buf.build/connectrpc/go
    out: gen
    opt: paths=source_relative
inputs:
  - directory: proto

Finally, regenerate your code:

$ buf generate

Now, your gen/ directory should look like this:

gen
└── pet
    └── v1
        ├── pet.pb.go
        └── petv1connect
            └── pet.connect.go

5. Implement the API

The Buf Schema Registry provides generated SDKs that allow you to import your schemas using standard package managers like go get, NPM, etc. instead of manually generating code. In this section, you'll implement a PetStoreService client by using the generated SDKs from your module. You can view all of your module's generated SDK options in the SDKs tab of its repository.

Fetch generated SDKs

First, fetch a few packages for the client you're building. In this example, you'll get the Go base types in protocolbuffers/go and Connect API stubs in connectrpc/go.

$ go get buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go
$ go get buf.build/gen/go/<USERNAME>/petapis/connectrpc/go

Implement the client

The server code has already been provided in the server/ folder, so start implementing a client by creating a client/main.go file:

$ mkdir client
$ touch client/main.go

Copy and paste this content into that file:

package main

import (
  "context"
  "log"
  "net/http"

  // Replace <USERNAME> with your BSR username if username isn't present
  "buf.build/gen/go/<USERNAME>/petapis/connectrpc/go/pet/v1/petv1connect"
  petv1 "buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go/pet/v1"
  connect "connectrpc.com/connect"
)

func main() {
  client := petv1connect.NewPetStoreServiceClient(
    http.DefaultClient,
    "http://localhost:8080",
  )
  res, err := client.PutPet(
    context.Background(),
    connect.NewRequest(&petv1.PutPetRequest{
      PetType: petv1.PetType_PET_TYPE_SNAKE,
      Name:    "Ekans",
    }),
  )
  if err != nil {
    log.Println(err)
    return
  }
  log.Println(res.Msg,"Successfully PutPet")
}

Resolve Go dependencies

Now that you have code for both a client and a server, you need to resolve some of the Go dependencies for the generated code:

$ go mod tidy

Your go.mod file should look something like this (the version pins may differ):

module github.com/bufbuild/buf-tour

go 1.19

require (
	buf.build/gen/go/<USERNAME>/petapis/connectrpc/go v1.16.1-20240424002215-e3c35074dc9a.1
	buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go v1.33.0-20240424002215-e3c35074dc9a.1
	connectrpc.com/connect v1.16.1
	github.com/bufbuild/connect-go v1.5.1
	golang.org/x/net v0.22.0
	google.golang.org/genproto v0.0.0-20240415180920-8c6c420018be
	google.golang.org/protobuf v1.33.0
)

require golang.org/x/text v0.14.0 // indirect

Call the endpoint

With the server/main.go and client/main.go implementations shown above, run the server and call the PutPet endpoint from the client.

First, run the server:

$ go run server/main.go
Output
... Listening on 127.0.0.1:8080

In a separate terminal, run the client and you should see a success message:

$ go run client/main.go
Output
2024/04/24 10:00:42  Successfully PutPet

You'll also notice a response like this in the server logs (in the other terminal running the server):

Output
2023/08/02 11:42:45 Got a request to create a PET_TYPE_SNAKE named Ekans

That's it! that's all you need to do to build a module, publish it, and build an API server and client using Buf.

Related docs

To find out more about how you can build better with Buf, check out some of our other guides:

• Getting started with the Buf CLI
• Modules and workspaces concepts guide
• Repositories concepts guide
• Dependency management in the BSR