Tutorial - buf build

We highly recommend completing the tour to get an introduction to buf build.

The operations on buf are based on Protobuf files that are built or compiled. The linter, breaking change detector, generator, and BSR features depend on the results of the compilation. The buf build command, in its simplest form, is used to verify that input successfully compiles.

Key concepts

buf is configured using the buf.yaml configuration file placed at the root of the Protobuf source files it defines. The placement of the buf.yaml configuration file tells buf where to search for .proto files and how to handle imports. buf recursively discovers all .proto files under configuration and builds them, unlike protoc, where all .proto files are manually specified on the command-line.

There's only one configuration option that affects the buf build command:

Default buf.yaml
version: v2
  - path: foo
      - foo/bar

The excludes option lists module subdirectories to ignore from .proto file discovery. Any directories added to this list are completely skipped and excluded from the result. The value of excludes must be relative to the workspace, not the module, since buf build is executed against the workspace. The example buf.yaml above is correct for a file structure that looks like this:

├── buf.yaml
└── foo
    └── bar

We recommend against using this option, but in some situations it may be unavoidable.

1. Define a workspace

To create a workspace and define the modules within it, add a buf.yaml file to the directory that contains your directories of .proto files. You can create the default buf.yaml file by running this command:

$ buf config init
Default buf.yaml
version: v2
    - FILE

Add a modules.path declaration for each directory you want to treat as a module. The value should be relative to the workspace root. See the buf.yaml documentation for more configuration details.

Workspace requirements

There are two additional requirements that buf imposes on your .proto file structure for compilation to succeed that are not enforced by protoc, both of which are essential to successful modern Protobuf development across a number of languages.

1. Workspace modules must not overlap—that is, one workspace module can't be a subdirectory of another workspace module.

This, for example, is not a valid configuration:

version: v2
  - path: foo
  - path: foo/bar

This is important to make sure that across all your .proto files, imports are consistent. In the above example, for a given file foo/bar/bar.proto, it would be valid to import this file as either bar/bar.proto or bar.proto. Having inconsistent imports leads to a number of major issues across the Protobuf plugin ecosystem.

All .proto file paths must be unique relative to each workspace module.

For example, given this configuration:

version: v2
  - path: foo
  - path: bar

it's invalid to have these two files:

  • foo/baz/baz.proto
  • bar/baz/baz.proto

This results in two files having the path baz/baz.proto. Imagine that a third file is thrown into the mix:

syntax = "proto3";

package bar.baz;

import "baz/baz.proto";

Which file is being imported here? Is it foo/baz/baz.proto? bar/baz/baz.proto? The answer depends on the order of the -I flags given to protoc, or (if buf didn't error in this scenario pre-compilation, which buf does) the order of the imports given to the internal compiler.

While the above example is relatively contrived, the common error that comes up is when you have vendored .proto files. For example, grpc-gateway has its own copy of the google.api definitions it needs. While these are usually in sync, the google.api schema can change. Imagine that we allowed this:

version: v2
  - path: proto
  - path: vendor/github.com/googleapis/googleapis
  - path: vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis

Which copy of google/api/*.proto wins? The answer: no one wins. So Buf doesn't allow this.

2. Run build

You can run buf build on your workspace by specifying the filepath to the directory containing the buf.yaml configuration file. To target the module defined in the current directory:

$ buf build

The buf build command:

  • Discovers all Protobuf files per your buf.yaml configuration.
  • Copies the Protobuf files into memory.
  • Compiles all Protobuf files.
  • Outputs the compiled result to a configurable location (defaults to /dev/null)

If there are errors, they are printed out in a file:line:column:message format by default. For example:

$ buf build
acme/pet/v1/pet.proto:5:8:acme/payment/v1alpha1/payment.proto: does not exist

Build output can also be printed as JSON:

$ buf build --error-format=json
{"path":"acme/pet/v1/pet.proto","start_line":5,"start_column":8,"end_line":5,"end_column":8,"type":"COMPILE","message":"acme/payment/v1alpha1/payment.proto: does not exist"}

3. Options

Output format

By default, buf build outputs its result to /dev/null. In this case, it's common to use buf build as a validation step, analogous to checking if the input compiles.

buf build also supports outputting FileDescriptorSet s and images, which are Buf's custom extension of the FileDescriptorSet. Better yet, these outputs can be formatted in a variety of ways.

buf build can deduce the output format by the file extension, see the documentation on automatically derived formats. For example,

$ buf build -o image.binpb
$ buf build -o image.binpb.gz
$ buf build -o image.binpb.zst
$ buf build -o image.json
$ buf build -o image.json.gz
$ buf build -o image.json.zst
$ buf build -o image.txtpb
$ buf build -o image.txtpb.gz
$ buf build -o image.txtpb.zst

The special value - is used to denote stdout, and you can manually set the format. For example:

$ buf build -o -#format=json

When combined with jq, buf build also allows for introspection. For example, to see a list of all packages, you can run this command:

$ buf build -o -#format=json | jq '.file[] | .package' | sort | uniq | head
"google.actions.type" "google.ads.admob.v1" "google.ads.googleads.v1.common" "google.ads.googleads.v1.enums" "google.ads.googleads.v1.errors" "google.ads.googleads.v1.resources" "google.ads.googleads.v1.services" "google.ads.googleads.v2.common" "google.ads.googleads.v2.enums" "google.ads.googleads.v2.errors"

Images always include the ImageExtension field. But if you want a pure FileDescriptorSet without this field set, and thus to mimic protoc entirely, you can use the --as-file-descriptor-set flag:

$ buf build -o image.binpb --as-file-descriptor-set

The ImageExtension field doesn't affect Protobuf plugins or any other operations, as they merely see this as an unknown field. But we provide the option in case you want it.

Limit to specific files

By default, buf builds all files under the buf.yaml configuration file. You can instead manually specify the file or directory paths to build. This is an advanced feature intended to be used for editor or Bazel integration—it's better to let buf discover all files under management and handle this for you.

The compiled result is limited to the given files if the --path flag is specified, as in this command:

$ buf build --path path/to/foo.proto --path path/to/bar.proto

Limit to specific types

When you run buf build to create a FileDescriptorSet or Buf image, the output contains all the Protobuf types declared in the module by default. But for some advanced use cases, you may want the image or FileDescriptorSet to contain only a subset of the types described in your Protobuf schemas.

Versions 1.1.0 and later of the buf CLI include a --type option for the buf build command that enables you to supply a fully qualified Protobuf name and limit the resulting image or FileDescriptorSet to only those descriptors required to represent those types and their required dependencies. This example usage restricts the output types to those required to represent pkg.foo.Bar:

$ buf build --type pkg.foo.Bar

The --type flag accepts fully qualified names for messages, enums, and services. These dependent descriptors are included in the build:

  • Messages
    • Messages and enums referenced in message fields
    • Any proto2 extension declarations for message fields
    • The parent message if this message is a nested definition
    • Any custom options for the message, its fields, and the file in which the message is defined
  • Enums
    • The enum value descriptors for this enum
    • The parent message is this enum is a nested definition
    • Any custom options for the enum, enum values, and the file in which the enum is defined
  • Services
    • Request and response types referenced in service methods
    • Any custom options for the services, its methods, and the file in which the service is defined
Supplying multiple types

You can specify multiple types by applying the --type option multiple times, as in this example:

$ buf build \
  --type acme.weather.v1.Units \
  --type acme.weather.v1.CurrentWeather.Temperature

In this case, dependent descriptors for both acme.weather.v1.Units and acme.weather.v1.CurrentWeather.Temperature are included in the output.

Type restriction example

As an example, consider these two .proto files:

package pkg;
message Foo {
  optional Bar bar = 1;
  extensions 2 to 3;
message Bar {...}
message Baz {
  other.Qux qux = 1 [(other.my_option).field = "buf"];
package other;
extend Foo {
  optional Qux baz = 2;
message Qux{...}
message Quux{...}
extend google.protobuf.FieldOptions {
  optional Quux my_option = 51234;

This table shows which files, messages, and extensions would be included for various types from foo.proto and bar.proto if specified as the argument to --type:

buf build --type pkg.Foofoo.proto, bar.protopkg.Foo, pkg.Bar, other.Quxother.baz\
buf build --type pkg.Barfoo.protopkg.Bar
buf build --type pkg.Bazfoo.proto, bar.protopkg.Baz, other.Quux, other.Quxother.my_option

5. Docker

Buf ships a Docker image bufbuild/buf that enables you to use buf as part of your Docker workflow. For example:

$ docker run \
  --volume "$(pwd):/workspace" \
  --workdir /workspace \
  bufbuild/buf build