This tour demonstrates the workflow and features of the Buf Schema Registry (BSR). You'll create a BSR repository, push a module of Protobuf files, and learn how it handles dependency management and generating API documentation. Finally, you will put it all together to build an API client using SDKs that the BSR auto-generates from the module.
Prerequisites
-
We assume you've already installed the Buf CLI,
git
andgo
in your$PATH
. If you haven't, go to our installation guide first. -
Check that the locally installed version of the Buf CLI is up to date.
$ buf --version
Output1.26.1 -
If you haven't already, sign up for a Buf account. A Buf Account grants you access to Buf Schema Registry repositories and allows you to explore modules that are available from the community and verified publishers. You'll also need a Buf account to share modules on the Buf Schema Registry.
Throughout this tour, we've referenced your BSR username in code samples to make copy/paste easier. If you're not logged
in, it appears as <USERNAME>
.
Clone the Git repository
To finish setting up, clone the Git repository that contains the starter code for the PetStore
service.
$ git clone https://github.com/bufbuild/buf-tour
In the start/getting-started-with-bsr
directory, there's a pre-configured module that defines the pet store API (which
specifies a way to create, get, and delete pets in the store), and some generated code to build an API server and
client.
We'll start in the start/getting-started-with-bsr
directory, and at the end of the tour, it should match the
finish/getting-started-with-bsr
directory.
$ cd buf-tour/start/getting-started-with-bsr
1. Create token and login
1.1. Create an API token
-
You'll be prompted with a few different login options, including Google, GitHub, and traditional email and password. You've successfully logged in if your username is shown in the upper right-hand corner.
-
Visit the buf.build/settings/user page.
-
Click the Create New Token button.
-
Select an expiration time. You can optionally add a note for yourself to distinguish this token from others (we recommend that you name this
CLI
,Development
, or something else along those lines). -
Click Create.
-
Copy the token to your clipboard. You'll use it for the rest of the tour.
1.2. Log in from the command line
Use the API token created above to log in:
$ buf registry login
OutputLog in with your Buf Schema Registry username. If you don't have a username, create one at https://buf.build. Username: <USERNAME> Token: <YOUR TOKEN>
2. Push a module
Now that you've authenticated with the BSR, you can create a repository and push the provided module that defines the
PetStoreService
API.
Modules are the core primitive of Buf and the BSR. A module is a collection of Protobuf files that are configured,
built, and versioned as a logical unit. The buf.yaml
file in the proto
folder of the tour repo initializes the
module, and the pet/v1/pet.proto
is the API schema.
2.1. Create a BSR repository
A module is stored in the BSR as a repository. A repository stores all versions of a module, where each version is identified by a commit, (optionally) a tag, and/or (optionally) a draft. Though roughly analogous to a Git repository, a BSR repository is only a remote location—there is no concept of a repository "clone". In other words, repositories do not exist in multiple locations.
To create a new repository:
- Navigate to the home page.
- Select your username in the top right corner.
- Select Repositories from the dropdown.
- Click the Create repository button.
- Name the repository
petapis
. For the purposes of this guide, keep the repository public.
You should see an empty repository called petapis
. Next up, you'll push a module.
2.2. Configure the repository name
Back in your terminal, move into the proto
directory:
$ cd proto
Add a name
field to your buf.yaml
file that matches the repository you just created:
version: v1
name: buf.build/<USERNAME>/petapis
breaking:
use:
- FILE
lint:
use:
- DEFAULT
2.3. Push the module
From the proto
directory, push the module to the buf.build/<USERNAME>/petapis
repository:
$ buf push
Output19bcefa1a736428d9e64d21c9191b213
The pushed module creates a commit ID, which is returned to the CLI. Your value will differ.
Behind the scenes, the Buf CLI recognizes the name
in your buf.yaml
and pushes the module to the
buf.build/<USERNAME>/petapis
repository. If successful, the generated commit identifies this current version of your
module.
3. View and edit documentation
When you push a module to the BSR, it automatically generates documentation for all messages, fields, enums and services
defined in your .proto
files. You can browse the generated documentation for your module by going to the Docs page
in your repository at https://buf.build/
3.1. Add module documentation
The page you see above serves as the primary entry point for your module's documentation. But as you can see from the default content, you currently don't have any module-level documentation.
You can update the module-level documentation page by creating a buf.md
in the same directory as your module's
buf.yaml
file, and pushing it to the BSR (the buf.md
file is analogous to a GitHub repository's README.md
file).
The buf.md
file supports all CommonMark syntax.
Start by adding a quick note:
$ touch buf.md
## PetAPIs
This module contains all the APIs required to interact with the `PetStoreService`.
Your proto
directory should now look like this:
proto/
├── buf.md
├── buf.yaml
├── google
│  └── type
│  └── datetime.proto
└── pet
└── v1
└── pet.proto
When you push the module again, there's a new commit, and if you refresh the documentation page you visited above, you should see the changes you just introduced.
$ buf push
Output4514ddced0584e73a100e82096c7958c
3.2. View package documentation
As you can see from the module documentation page, both the pet.v1
and google.type
packages are available as links.
Click on the pet.v1
link to navigate to its package documentation. From here, you can click through each of the
Protobuf type definitions and see all the comments associated with each type. In fact, if you click on the
google.type.DateTime
message referenced in the Pet
message, you'll be brought to the google.type.v1
package
documentation for the same commit.
For another example of API documentation, check out buf.build/googleapis/googleapis.
4. Add a dependency
Without the BSR, you can only depend on other Protobuf APIs by manually fetching the .proto
files you need. For
example, if you want to use googleapis
, you need to clone the right Git
repository and copy the .proto
files locally to compile your own .proto
files. And if googleapis
has its own
external dependencies, you need to fetch those as well.
This way of managing dependencies is prone to API drift. If the googleapis
code evolves over time, your local copies
are inconsistent with the latest version and your modules become out of date. In the example code, it turns out that
this is exactly what happened: the google/type/datetime.proto
file is present in your local directory and used to
build your module.
Now that you're familiar with the BSR, you can simplify this entire workflow immensely.
4.1. Remove the datetime.proto
file
Start by removing the google/type/datetime.proto
file from your module . From within the proto
directory, run this
command to remove the local google
dependencies:
$ rm -r google
Now remove the google/type/datetime.proto
reference from your buf.yaml
file:
version: v1
name: buf.build/<USERNAME>/petapis
breaking:
use:
- FILE
lint:
use:
- DEFAULT
- ignore:
- - google/type/datetime.proto
If you try to build the module in its current state, you get an error:
$ buf build
Outputpet/v1/pet.proto:7:8:google/type/datetime.proto: does not exist
4.2. Depend on googleapis
You resolve this error by adding a dependency in your buf.yaml
file's deps
key. The google/type/datetime.proto
file available in the BSR-hosted buf.build/googleapis/googleapis
module, so you can configure it like this:
version: v1
name: buf.build/<USERNAME>/petapis
+deps:
+ - buf.build/googleapis/googleapis
breaking:
use:
- FILE
lint:
use:
- DEFAULT
When you try to build the module again, you'll get this warning:
$ buf build
OutputWARN Specified deps are not covered in your buf.lock, run "buf mod update": - buf.build/googleapis/googleapis pet/v1/pet.proto:7:8:google/type/datetime.proto: does not exist
buf
detected that you specified a dependency that isn't included in the module's buf.lock
file. This file is a
dependency manifest for your module, representing a single reproducible build of your module's dependencies. You don't
have a buf.lock
file yet because you haven't specified any external dependencies, but you can create one with the
command that buf
recommended:
$ buf mod update
The buf mod update
command updates all of your deps
to their latest version. The generated buf.lock
file should
look similar to this (the commit
ID will vary):
# Generated by buf. DO NOT EDIT.
version: v1
deps:
- remote: buf.build
owner: googleapis
repository: googleapis
commit: 62f35d8aed1149c291d606d958a7ce32
When you try to build the module again, it's successful:
$ buf build
Outputbuf: downloading buf.build/googleapis/googleapis:62f35d8aed1149c291d606d958a7ce32
This is the BSR's dependency management in action! A few things happened here, so let's break it down:
buf
noticed that a new dependency was added to thedeps
key.buf
resolved the latest version of thebuf.build/googleapis/googleapis
module and wrote it to the module'sbuf.lock
file.- When another
buf
command is run,buf
downloaded thebuf.build/googleapis/googleapis
module to the local module cache. - Finally, now that
buf
has all the dependencies it needs, it successfully builds the module (becausegoogle/type/datetime.proto
is included).
In summary, buf
can resolve the dependencies specified in your buf.yaml
's deps
key and include the imports
required to build your module. You don't have to manually copy .proto
files any more!
4.3. Push your changes
Now that you've updated your module to depend on buf.build/googleapis/googleapis
instead of vendoring the
google/type/datetime.proto
yourself, you can push the module to the BSR:
$ buf push
Outputb2917eb692064beb92ad1e38dba6c25e
Navigate back to your repository on the BSR and you'll notice your packages have changed. The Google package is no longer a first-class citizen of your module—it's now a third-party dependency.
If one or more dependencies are pinned to a draft commit, pushing the module is not allowed.
4.4. Update the buf.gen.yaml
file and regenerate code
Your gen/
directory should look like this
gen
├── google
│  └── type
│  └── datetime.pb.go
└── pet
└── v1
├── pet.pb.go
└── petv1connect
└── pet.connect.go
Now that you've exchanged your local copy of the Google proto for the one on the BSR, you can remove the generated code
also. First, remove the gen/
directory:
$ cd ..
$ rm -r gen
Then update the buf.gen.yaml
to exclude overriding any Go import statements related to googleapis
.
version: v1
managed:
enabled: true
go_package_prefix:
default: github.com/bufbuild/buf-tour/petstore/gen
+ except:
+ - buf.build/googleapis/googleapis
plugins:
- plugin: buf.build/protocolbuffers/go
out: gen
opt: paths=source_relative
- plugin: buf.build/connectrpc/go
out: gen
opt: paths=source_relative
Finally, regenerate your code:
$ buf generate proto
Now, your gen/
directory should look like this:
gen
└── pet
└── v1
├── pet.pb.go
└── petv1connect
└── pet.connect.go
5. Implement the API
In this section, you'll implement a PetStoreService
client that you can run on the command line.
5.1. Fetch remote packages
The Buf Schema Registry provides remote packages for Go using go get
, just like any other Go library. This means that
if you use the BSR, there's no need to manage generated code. You can view all of your module's remote package options
in the Assets tab of its repository.
First, fetch a few packages for the client you're building. In this example, you'll get the Go base-types in
protocolbuffers/go
and Connect API stubs in connectrpc/go
.
$ go get buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go
$ go get buf.build/gen/go/<USERNAME>/petapis/connectrpc/go
5.3. Implement the client
The server code has already been provided in the server/
folder, so start implementing a client by creating a
client/main.go
file:
$ mkdir client
$ touch client/main.go
Copy and paste this content into that file:
package main
import (
"context"
"log"
"net/http"
// Replace <USERNAME> with your BSR username if username isn't present
"buf.build/gen/go/<USERNAME>/petapis/connectrpc/go/pet/v1/petv1connect"
petv1 "buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go/pet/v1"
connect "connectrpc.com/connect"
)
func main() {
client := petv1connect.NewPetStoreServiceClient(
http.DefaultClient,
"http://localhost:8080",
)
res, err := client.PutPet(
context.Background(),
connect.NewRequest(&petv1.PutPetRequest{
PetType: petv1.PetType_PET_TYPE_SNAKE,
Name: "Ekans",
}),
)
if err != nil {
log.Println(err)
return
}
log.Println(res.Msg)
}
5.4. Resolve Go dependencies
Now that you have code for both a client and a server, run this command to resolve some of the dependencies you need for the generated code:
$ go mod tidy
You should see these changes (the version pins may differ):
module github.com/bufbuild/buf-tour/petstore
go 1.19
require (
buf.build/gen/go/<USERNAME>/petapis/connectrpc/go v1.11.0-20230203192357-a60a321c3624.1
buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go v1.28.1-20230203192357-a60a321c3624.4
github.com/connectrpc/connect-go v1.11.0
golang.org/x/net v0.5.0
google.golang.org/genproto v0.0.0-20230202175211-008b39050e57
google.golang.org/protobuf v1.28.1
)
require golang.org/x/text v0.6.0 // indirect
5.5. Call PutPet
With the server/main.go
and client/main.go
implementations shown above, run the server and call the PutPet
endpoint from the client.
First, run the server:
$ go run server/main.go
Output... Listening on 127.0.0.1:8080
In a separate terminal, run the client and you should see a success message:
$ go run client/main.go
Output... Connected to 127.0.0.1:8080 ... Successfully PutPet
You'll also notice a response like this in the server logs (in the other terminal running the server):
... Listening on 127.0.0.1:8080
2023/08/02 11:42:45 Got a request to create a PET_TYPE_SNAKE named Ekans
That's it! that's all you need to do to build a module, publish it, and build an API server and client using Buf.
Next steps
To find out more about how you can build better with Buf, check out some of our other guides: