Try the Buf Schema Registry

This tour demonstrates the workflow and features of the Buf Schema Registry (BSR). You'll create a BSR repository, push a module of Protobuf files, and learn how it handles dependency management and generating API documentation. Finally, you will put it all together to build an API client using SDKs that the BSR auto-generates from the module.

Prerequisites

• We assume you've already installed the Buf CLI, git and go in your $PATH. If you haven't, go to our installation guide first.

• Check that the locally installed version of the Buf CLI is up to date.

  $ buf --version
  Output
  1.28.1
  

• If you haven't already, sign up for a Buf account. A Buf Account grants you access to Buf Schema Registry repositories and allows you to explore modules that are available from the community and verified publishers. You'll also need a Buf account to share modules on the Buf Schema Registry.

Clone the Git repository

To finish setting up, clone the Git repository that contains the starter code for the PetStore service.

$ git clone https://github.com/bufbuild/buf-tour

In the start/getting-started-with-bsr directory, there's a pre-configured module that defines the pet store API (which specifies a way to create, get, and delete pets in the store), and some generated code to build an API server and client.

We'll start in the start/getting-started-with-bsr directory, and at the end of the tour, it should match the finish/getting-started-with-bsr directory.

$ cd buf-tour/start/getting-started-with-bsr

1. Create token and login

1.1. Create an API token

1. Log in to the BSR.

   You'll be prompted with a few different login options, including Google, GitHub, and traditional email and password. You've successfully logged in if your username is shown in the upper right-hand corner.

2. Visit the buf.build/settings/user page.

3. Click the Create New Token button.

4. Select an expiration time. You can optionally add a note for yourself to distinguish this token from others (we recommend that you name this CLI, Development, or something else along those lines).

5. Click Create.

6. Copy the token to your clipboard. You'll use it for the rest of the tour.

1.2. Log in from the command line

Use the API token created above to log in:

$ buf registry login
Output
Log in with your Buf Schema Registry username. If you don't have a username, create one at https://buf.build.

Username: <USERNAME>
Token: <YOUR TOKEN>

2. Push a module

Now that you've authenticated with the BSR, you can create a repository and push the provided module that defines the PetStoreService API.

Modules are the core primitive of Buf and the BSR. A module is a collection of Protobuf files that are configured, built, and versioned as a logical unit. The buf.yaml file in the proto folder of the tour repo initializes the module, and the pet/v1/pet.proto is the API schema.

2.1. Create a BSR repository

A module is stored in the BSR as a repository. A repository stores all versions of a module, where each version is identified by a commit, (optionally) a tag, and/or (optionally) a branch. Though roughly analogous to a Git repository, a BSR repository is only a remote location—there is no concept of a repository "clone". In other words, repositories do not exist in multiple locations.

To create a new repository:

1. Navigate to the home page.
2. Select your username in the top right corner.
3. Select Repositories from the dropdown.
4. Click the Create repository button.
5. Name the repository petapis. For the purposes of this guide, keep the repository public.

You should see an empty repository called petapis. Next up, you'll push a module. (For the purposes of this tutorial, please skip the suggested commands on the screen and continue to step 2.2.)

2.2. Configure the repository name

Back in your terminal, move into the proto directory:

$ cd proto

Add a name field to your buf.yaml file that matches the repository you just created:

version: v1
name: buf.build/<USERNAME>/petapis
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

2.3. Push the module

From the proto directory, push the module to the buf.build/<USERNAME>/petapis repository:

$ buf push
Output
19bcefa1a736428d9e64d21c9191b213

The pushed module creates a commit ID, which is returned to the CLI. Your value will differ.

Behind the scenes, the Buf CLI recognizes the name in your buf.yaml and pushes the module to the buf.build/<USERNAME>/petapis repository. If successful, the generated commit identifies this current version of your module.

3. View and edit documentation

When you push a module to the BSR, it automatically generates documentation for all messages, fields, enums and services defined in your .proto files. You can browse the generated documentation for your module by going to the Docs page in your repository at https://buf.build/

3.1. Add module documentation

The page you see above serves as the primary entry point for your module's documentation. But as you can see from the default content, you currently don't have any module-level documentation.

You can update the module-level documentation page by creating a buf.md in the same directory as your module's buf.yaml file, and pushing it to the BSR (the buf.md file is analogous to a GitHub repository's README.md file). The buf.md file supports all CommonMark syntax.

Start by adding a quick note:

$ touch buf.md

## PetAPIs

This module contains all the APIs required to interact with the `PetStoreService`.

Your proto directory should now look like this:

proto/
├── buf.md
├── buf.yaml
├── google
│   └── type
│       └── datetime.proto
└── pet
    └── v1
        └── pet.proto

When you push the module again, there's a new commit, and if you refresh the documentation page you visited above, you should see the changes you just introduced.

$ buf push
Output
4514ddced0584e73a100e82096c7958c

3.2. View package documentation

As you can see from the module documentation page, both the pet.v1 and google.type packages are available as links. Click on the pet.v1 link to navigate to its package documentation. From here, you can click through each of the Protobuf type definitions and see all the comments associated with each type. In fact, if you click on the google.type.DateTime message referenced in the Pet message, you'll be brought to the google.type.v1 package documentation for the same commit.

4. Add a dependency

Without the BSR, you can only depend on other Protobuf APIs by manually fetching the .proto files you need. For example, if you want to use googleapis, you need to clone the right Git repository and copy the .proto files locally to compile your own .proto files. And if googleapis has its own external dependencies, you need to fetch those as well.

This way of managing dependencies is prone to API drift. If the googleapis code evolves over time, your local copies are inconsistent with the latest version and your modules become out of date. In the example code, it turns out that this is exactly what happened: the google/type/datetime.proto file is present in your local directory and used to build your module.

Now that you're familiar with the BSR, you can simplify this entire workflow immensely.

4.1. Remove the datetime.proto file

Start by removing the google/type/datetime.proto file from your module. From within the proto directory, run this command to remove the local google dependencies:

$ rm -r google

Now remove the google/type/datetime.proto reference from your buf.yaml file:

 version: v1
 name: buf.build/<USERNAME>/petapis
 breaking:
   use:
     - FILE
 lint:
   use:
     - DEFAULT
 -  ignore:
 -    - google/type/datetime.proto

If you try to build the module in its current state, you get an error:

$ buf build
Output
pet/v1/pet.proto:7:8:google/type/datetime.proto: does not exist

4.2. Depend on googleapis

You resolve this error by adding a dependency in your buf.yaml file's deps key. The google/type/datetime.proto file available in the BSR-hosted buf.build/googleapis/googleapis module, so you can configure it like this:

 version: v1
 name: buf.build/<USERNAME>/petapis
 +deps:
 +  - buf.build/googleapis/googleapis
 breaking:
   use:
     - FILE
 lint:
   use:
     - DEFAULT

When you try to build the module again, you'll get this warning:

$ buf build
Output
WARN  Specified deps are not covered in your buf.lock, run "buf mod update":
  - buf.build/googleapis/googleapis
pet/v1/pet.proto:7:8:google/type/datetime.proto: does not exist

buf detected that you specified a dependency that isn't included in the module's buf.lock file. This file is a dependency manifest for your module, representing a single reproducible build of your module's dependencies. You don't have a buf.lock file yet because you haven't specified any external dependencies, but you can create one with the command that buf recommended:

$ buf mod update

The buf mod update command updates all of your deps to their latest version. The generated buf.lock file should look similar to this (the commit ID and digest will vary):

# Generated by buf. DO NOT EDIT.
version: v1
deps:
  - remote: buf.build
    owner: googleapis
    repository: googleapis
    commit: 62f35d8aed1149c291d606d958a7ce32
    digest: shake256:492e5hf8ef57f7863e6f22004539deff8834cf39de

When you try to build the module again, it's successful:

$ buf build

This is the BSR's dependency management in action! A few things happened here, so let's break it down:

1. buf noticed that a new dependency was added to the deps key.
2. buf resolved the latest version of the buf.build/googleapis/googleapis module and wrote it to the module's buf.lock file.
3. When another buf command is run, buf downloaded the buf.build/googleapis/googleapis module to the local module cache.
4. Finally, now that buf has all the dependencies it needs, it successfully builds the module (because google/type/datetime.proto is included).

In summary, buf can resolve the dependencies specified in your buf.yaml's deps key and include the imports required to build your module. You don't have to manually copy .proto files any more!

4.3. Push your changes

Now that you've updated your module to depend on buf.build/googleapis/googleapis instead of vendoring the google/type/datetime.proto yourself, you can push the module to the BSR:

$ buf push
Output
b2917eb692064beb92ad1e38dba6c25e

Navigate back to your repository on the BSR and you'll notice your packages have changed. The Google package is no longer a first-class citizen of your module—it's now a third-party dependency.

4.4. Update the buf.gen.yaml file and regenerate code

Your gen/ directory should look like this

gen
├── google
│   └── type
│       └── datetime.pb.go
└── pet
    └── v1
        ├── pet.pb.go
        └── petv1connect
            └── pet.connect.go

Now that you've exchanged your local copy of the Google proto for the one on the BSR, you can remove the generated code also. First, remove the gen/ directory:

$ cd ..
$ rm -r gen

Then update the buf.gen.yaml to exclude overriding any Go import statements related to googleapis.

 version: v1
 managed:
   enabled: true
   go_package_prefix:
     default: github.com/bufbuild/buf-tour/petstore/gen
 +    except:
 +      - buf.build/googleapis/googleapis
 plugins:
   - plugin: buf.build/protocolbuffers/go
     out: gen
     opt: paths=source_relative
   - plugin: buf.build/connectrpc/go
     out: gen
     opt: paths=source_relative

Finally, regenerate your code:

$ buf generate proto

Now, your gen/ directory should look like this:

gen
└── pet
    └── v1
        ├── pet.pb.go
        └── petv1connect
            └── pet.connect.go

5. Implement the API

In this section, you'll implement a PetStoreService client that you can run on the command line.

5.1. Fetch generated SDKs

The Buf Schema Registry provides generated SDKs for Go using go get, just like any other Go library. This means that if you use the BSR, there's no need to manage generated code. You can view all of your module's generated SDK options in the SDKs tab of its repository.

First, fetch a few packages for the client you're building. In this example, you'll get the Go base-types in protocolbuffers/go and Connect API stubs in connectrpc/go.

$ go get buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go
$ go get buf.build/gen/go/<USERNAME>/petapis/connectrpc/go

5.3. Implement the client

The server code has already been provided in the server/ folder, so start implementing a client by creating a client/main.go file:

$ mkdir client
$ touch client/main.go

Copy and paste this content into that file:

package main

import (
  "context"
  "log"
  "net/http"

  // Replace <USERNAME> with your BSR username if username isn't present
  "buf.build/gen/go/<USERNAME>/petapis/connectrpc/go/pet/v1/petv1connect"
  petv1 "buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go/pet/v1"
  connect "connectrpc.com/connect"
)

func main() {
  client := petv1connect.NewPetStoreServiceClient(
    http.DefaultClient,
    "http://localhost:8080",
  )
  res, err := client.PutPet(
    context.Background(),
    connect.NewRequest(&petv1.PutPetRequest{
      PetType: petv1.PetType_PET_TYPE_SNAKE,
      Name:    "Ekans",
    }),
  )
  if err != nil {
    log.Println(err)
    return
  }
  log.Println(res.Msg)
}

5.4. Resolve Go dependencies

Now that you have code for both a client and a server, run this command to resolve some of the dependencies you need for the generated code:

$ go mod tidy

You should see these changes (the version pins may differ):

module github.com/bufbuild/buf-tour/petstore

go 1.19

require (
  buf.build/gen/go/<USERNAME>/petapis/connectrpc/go v1.11.0-20230203192357-a60a321c3624.1
  buf.build/gen/go/<USERNAME>/petapis/protocolbuffers/go v1.28.1-20230203192357-a60a321c3624.4
  github.com/connectrpc/connect-go v1.11.0
  golang.org/x/net v0.5.0
  google.golang.org/genproto v0.0.0-20230202175211-008b39050e57
  google.golang.org/protobuf v1.28.1
)

require golang.org/x/text v0.6.0 // indirect

5.5. Call PutPet

With the server/main.go and client/main.go implementations shown above, run the server and call the PutPet endpoint from the client.

First, run the server:

$ go run server/main.go
Output
... Listening on 127.0.0.1:8080

In a separate terminal, run the client and you should see a success message:

$ go run client/main.go
Output
... Connected to 127.0.0.1:8080
... Successfully PutPet

You'll also notice a response like this in the server logs (in the other terminal running the server):

... Listening on 127.0.0.1:8080
2023/08/02 11:42:45 Got a request to create a PET_TYPE_SNAKE named Ekans

That's it! that's all you need to do to build a module, publish it, and build an API server and client using Buf.

Next steps

To find out more about how you can build better with Buf, check out some of our other guides:

• Getting Started with the Buf CLI
• Modules overview
• Repositories overview
• Dependency management