Try the Buf Schema Registry
This tour demonstrates the workflow and features of the Buf Schema Registry (BSR), which helps organizations manage schemas, dependencies, and governance at scale. You'll create a BSR repository, push a module of Protobuf files, and learn how it handles dependency management and generating API documentation. Finally, you'll put it all together to build an API client using SDKs that the BSR auto-generates from the module.
Prerequisites
-
Install the Buf CLI if you haven't already. You need version 1.32 or higher to do the tour, so if you previously installed the Buf CLI, check the version and update if necessary:
-
Clone the
buf-tourrepo: -
Sign up for a Buf account if you haven't already. A Buf account is required to access and share BSR repositories and allows you to explore modules that are available from the community and verified publishers.
The repository contains a start directory and a finish directory.
During this tour you'll work on files in the start/getting-started-with-bsr directory, and at the end they should match the files in the
finish/getting-started-with-bsr directory.
1. Create an API token and sign in
Create a token
-
Sign in to the BSR. You'll be prompted with a few options, including Google, GitHub, and traditional email and password. You've successfully logged in if your username is shown in the upper right-hand corner.
-
Go to the User settings page.
- Click the Create New Token button.
- Select an expiration time. You can optionally add a note for yourself to distinguish this token from others.
- Click Create.
- Copy the token to your clipboard—you'll use it for the rest of the tour.
Sign in from the command line
Use the API token you created to sign in:
$ buf registry login
Log in with your Buf Schema Registry username. If you don't have a username, create one at https://buf.build.
Username: USERNAME
Token: YOUR_TOKEN
2. Push a module
Now that you've authenticated with the BSR, you can create a repository and push the provided module that defines the
PetStoreService API.
Modules are the core primitive of Buf and the BSR.
A module is a collection of Protobuf files that are configured, built, and versioned as a logical unit.
The buf.yaml file in the tour repo initializes the module, and the pet/v1/pet.proto is the API schema.
Create a BSR repository
A module is stored in the BSR as a repository. Repositories store all versions of a module, where each version is identified by a commit and (optionally) one or more labels. Though roughly analogous to a Git repository, a BSR repository is only a remote storage location—there is no concept of a repository "clone". BSR repositories don't exist in multiple locations.
To create a new repository:
- Go to the home page of the BSR.
- Select your username in the top right corner.
- Select Repositories from the dropdown.
- Click the Create repository button.
-
Name the repository
petapis. Keep the repository public.
You should see an empty repository named petapis.
Next up, you'll push a module.
For the purposes of this tutorial, please skip the suggested commands on the screen and continue to the next step.
Configure the module name
Next, you need to configure your local module (provided in the sample code) so that its history can be stored in the repository you just created.
This connection is made via the name field in the module's buf.yaml configuration file, which must include the full path to the repository (BSR instaance, organization, and repository name).
The buf.yaml file can list multiple modules, each of which corresponds to a single repository on the BSR.
This example only has one, which is contained in the proto subdirectory.
Add a name field to your buf.yaml file that matches the repository you just created:
version: v2
modules:
- path: proto
+ name: buf.build/USERNAME/petapis
lint:
use:
- STANDARD
ignore:
- proto/google/type/datetime.proto
breaking:
use:
- FILE
Push the module
Now push the module to the buf.build/USERNAME/petapis repository:
$ buf push
buf.build/USERNAME/petapis:4b28576bedf545cba5fa30ba741d1c6d
Pushing the module creates a commit in the BSR, which returns a commit ID to the CLI.
Behind the scenes, the Buf CLI recognizes the name in your buf.yaml and pushes the module to the buf.build/USERNAME/petapis repository.
If successful, the generated commit identifies this version of your
module.
3. View and edit documentation
When you push a module to the BSR, it automatically generates documentation for all messages, fields, enums, and services defined in your .proto files.
You can browse the generated documentation for your module by going to the Docs page in your repository at https://buf.build/USERNAME/petapis/docs.
Add module-level documentation
The page you see above serves as the primary entry point for your module's documentation. However, as you can see from the default content, you currently don't have any module-level documentation.
You can add it by creating a README.md file in the root directory of your module (proto in this example), and pushing the module to the BSR.
The README.md file is analogous to a GitHub repository's README.md file and supports all CommonMark syntax.
Create the file:
Then copy/paste this Markdown content into it to create an introductory note:
## PetAPIs
This module contains all the APIs required to interact with the `PetStoreService`.
Your proto directory should now look like this:
Push the module again, and if you refresh the documentation page you visited above, you should see the changes you just introduced.
View package documentation
As you can see from the module documentation page, both the pet.v1 and google.type packages are available as links.
Click on the pet.v1 link to navigate to its package documentation.
From here, you can click through each of the Protobuf type definitions and see all the comments associated with each type.
In fact, if you click on the google.type.DateTime message referenced in the Pet message, you'll be brought to the google.type.v1 package
documentation for the same commit.
4. Add a dependency
Without the BSR, you can only depend on other Protobuf APIs by manually fetching the .proto files you need.
For example, if you want to use googleapis, you need to clone the right Git
repository and copy the .proto files locally to compile your own .proto files.
And if googleapis has its own external dependencies, you need to fetch those as well.
This way of managing dependencies is prone to API drift.
If the googleapis code evolves over time, your local copies are inconsistent with the latest version and your modules become out of date.
In the example code, it turns out that this is exactly what happened: the google/type/datetime.proto file is present in your local directory and used to build your module.
Now that you're familiar with the BSR, you can simplify this entire workflow immensely.
Remove the datetime.proto file
Start by removing the google/type/datetime.proto file from your module.
From within the proto directory, run this command to remove the local google dependencies:
Now remove the google/type/datetime.proto reference from your buf.yaml file:
version: v2
modules:
- path: proto
name: buf.build/USERNAME/petapis
lint:
use:
- STANDARD
- ignore:
- - google/type/datetime.proto
breaking:
use:
- FILE
If you try to build the module in its current state, you get an error:
$ buf build
pet/v1/pet.proto:5:8:import "google/type/datetime.proto": file doesn't exist
Depend on googleapis
You resolve this error by adding a dependency in your buf.yaml file's deps key.
The google/type/datetime.proto
file is available in the BSR-hosted buf.build/googleapis/googleapis module, so you can configure it like this:
version: v2
modules:
- path: proto
name: buf.build/USERNAME/petapis
lint:
use:
- STANDARD
breaking:
use:
- FILE
+deps:
+ - buf.build/googleapis/googleapis
If you try to build the module again, you'll get the same warning, because even though you've specified the dependency, you don't yet have a buf.lock dependency manifest that pins the dependency to a specific version.
To create one, run this command:
The buf dep update command updates all of your deps to their latest version.
The generated buf.lock file should look similar to this (the commit ID and digest vary):
# Generated by buf. DO NOT EDIT.
version: v2
deps:
- name: buf.build/googleapis/googleapis
commit: 4ed3bc159a8b4ac68fe253218760d035
digest: b5:74a7798987b123218c004cf28543a2835e432ca04a69de99cd394a29dbad24d9ed38344f0b7c97ad6476039506c4eb38c2f4a8eef9cec3da2e38e4216a22d495
When you try to build the module again, it should be successful. A few things happened here, so let's break it down:
- The Buf CLI noticed that a new dependency was added to the
depskey. - It resolved the latest version of the
buf.build/googleapis/googleapismodule and wrote it to the module'sbuf.lockfile. - When you ran
buf buildagain , it downloaded thebuf.build/googleapis/googleapismodule to the local module cache. - Once all of the dependencies were available locally, it successfully built the module (because
google/type/datetime.protowas in the local module cache).
In summary, you can define your dependencies once in your buf.yaml file for all modules in the workspace, and the BSR resolves the dependencies to include the imports required to build the modules.
You don't have to manually copy .proto files any more!
Push your changes
Now that you've updated your module to depend on buf.build/googleapis/googleapis instead of vendoring the
google/type/datetime.proto yourself, you can push the module to the BSR:
$ buf push
buf.build/USERNAME/petapis:e3c35074dc9a43989d9f5c546a036d8b
Navigate back to your repository on the BSR and you'll notice your packages have changed. The Google package is no longer a first-class citizen of your module—it's now a third-party dependency. You can see the external dependencies for any module by going to its Deps tab:
This is what a dependency on googleapis looks like.
Note that it's pinned to a specific commit of googleapis—which for your module matches the one listed in your buf.lock file.
Update the buf.gen.yaml file and regenerate code
The tour repo includes generated code, so your gen/ directory should look like this:
gen
├── google
│ └── type
│ └── datetime.pb.go
└── pet
└── v1
├── pet.pb.go
└── petv1connect
└── pet.connect.go
Now that you've exchanged your local copy of googleapis for the one on the BSR, you can remove the generated code also.
First, remove the gen/ directory:
Then update the buf.gen.yaml to exclude overriding any Go import statements related to googleapis.
version: v2
managed:
enabled: true
+ disable:
+ - module: buf.build/googleapis/googleapis
override:
- file_option: go_package_prefix
value: github.com/bufbuild/buf-tour/gen
plugins:
- remote: buf.build/protocolbuffers/go
out: gen
opt: paths=source_relative
- remote: buf.build/connectrpc/go
out: gen
opt: paths=source_relative
inputs:
- directory: proto
Finally, regenerate your code:
Now, your gen/ directory should look like this:
5. Implement the API
The Buf Schema Registry provides generated SDKs that allow you to import your schemas using standard package managers like go get, NPM, etc. instead of manually generating code.
In this section, you'll implement a PetStoreService client by using the generated SDKs from your module.
You can view all of your module's generated SDK options in the SDKs tab of its repository.
Fetch generated SDKs
First, fetch a few packages for the client you're building.
In this example, you'll get the Go base types in protocolbuffers/go and Connect API stubs in connectrpc/go.
$ go get buf.build/gen/go/USERNAME/petapis/protocolbuffers/go
$ go get buf.build/gen/go/USERNAME/petapis/connectrpc/go
Implement the client
The server code has already been provided in the server/ folder, so start implementing a client by creating a client/main.go file:
Copy and paste this content into that file:
package main
import (
"context"
"log"
"net/http"
// Replace <USERNAME> with your BSR username if username isn't present
"buf.build/gen/go/xUSERNAMEx/petapis/connectrpc/go/pet/v1/petv1connect"
petv1 "buf.build/gen/go/xUSERNAMEx/petapis/protocolbuffers/go/pet/v1"
connect "connectrpc.com/connect"
)
func main() {
client := petv1connect.NewPetStoreServiceClient(
http.DefaultClient,
"http://localhost:8080",
)
res, err := client.PutPet(
context.Background(),
connect.NewRequest(&petv1.PutPetRequest{
PetType: petv1.PetType_PET_TYPE_SNAKE,
Name: "Ekans",
}),
)
if err != nil {
log.Println(err)
return
}
log.Println(res.Msg, "Successfully PutPet")
}
Resolve Go dependencies
Now that you have code for both a client and a server, you need to resolve some of the Go dependencies for the generated code:
Your go.mod file should look something like this (the version pins may differ):
module github.com/bufbuild/buf-tour
go 1.19
require (
buf.build/gen/go/USERNAME/petapis/connectrpc/go v1.16.1-20240424002215-e3c35074dc9a.1
buf.build/gen/go/USERNAME/petapis/protocolbuffers/go v1.33.0-20240424002215-e3c35074dc9a.1
connectrpc.com/connect v1.16.1
github.com/connectrpc/connect-go v1.5.1
golang.org/x/net v0.22.0
google.golang.org/genproto v0.0.0-20240415180920-8c6c420018be
google.golang.org/protobuf v1.33.0
)
require golang.org/x/text v0.14.0 // indirect
Call the endpoint
With the provided server/main.go and the client/main.go implementation shown above, run the server and call the PutPet
endpoint from the client.
First, run the server:
$ go run server/main.go
... Listening on 127.0.0.1:8080
In a separate terminal, run the client and you should see a success message:
$ go run client/main.go
2024/04/24 10:00:42 Successfully PutPet
You'll also notice a response like this in the server logs (in the other terminal running the server):
... Listening on 127.0.0.1:8080
2023/08/02 11:42:45 Got a request to create a PET_TYPE_SNAKE named Ekans
That's it! that's all you need to do to build a module, publish it, and build an API server and client using Buf.
Related docs
To find out more about how you can build better with Buf, check out some of our other guides:
- Getting started with the Buf CLI
- Modules and workspaces concepts guide
- Repositories concepts guide
- Dependency management in the BSR