Tutorial - buf build
We highly recommend completing the tour to get an introduction to
buf build
.
The operations on buf
are based on Protobuf files that are built or compiled. The linter, breaking change
detector, generator, and BSR features depend on the results of the
compilation. The buf build
command, in its simplest form, is used to verify that input successfully compiles.
Key concepts
buf
is configured using the buf.yaml
configuration file placed at the root of the Protobuf source files
it defines. The placement of the buf.yaml
configuration file tells buf where to search for .proto
files
and how to handle imports. buf
recursively discovers all .proto
files under configuration and builds them, unlike
protoc, where all .proto
files are manually specified on the command-line.
There's only one configuration option that affects the buf build
command:
version: v2
modules:
- path: foo
excludes:
- foo/bar
The excludes
option lists module subdirectories to ignore from .proto
file discovery. Any directories added to this
list are completely skipped and excluded from the result. The value of excludes
must be relative to the workspace,
not the module, since buf build
is executed against the workspace. The example buf.yaml
above is correct for a file
structure that looks like this:
workspace_root
├── buf.yaml
└── foo
└── bar
We recommend against using this option, but in some situations it may be unavoidable.
1. Define a workspace
To create a workspace and define the modules within it, add a buf.yaml
file to the directory that
contains your directories of .proto
files. You can create the default buf.yaml
file by running this command:
$ buf config init
version: v2
breaking:
use:
- FILE
lint:
use:
- DEFAULT
Add a modules.path
declaration for each directory you want to treat as a module. The value should be relative to the
workspace root. See the buf.yaml
documentation for more configuration details.
Workspace requirements
There are two additional requirements that buf
imposes on your .proto
file structure for compilation to succeed that
are not enforced by protoc
, both of which are essential to successful modern Protobuf development across a number of
languages.
1. Workspace modules must not overlap—that is, one workspace module can't be a subdirectory of another workspace module.
This, for example, is not a valid configuration:
version: v2
# THIS IS INVALID AND RESULTS IN A PRE-COMPILATION ERROR
modules:
- path: foo
- path: foo/bar
This is important to make sure that across all your .proto
files, imports are consistent. In the above example, for a
given file foo/bar/bar.proto
, it would be valid to import this file as either bar/bar.proto
or bar.proto
. Having
inconsistent imports leads to a number of major issues across the Protobuf plugin ecosystem.
All .proto
file paths must be unique relative to each workspace module.
For example, given this configuration:
version: v2
modules:
- path: foo
- path: bar
it's invalid to have these two files:
foo/baz/baz.proto
bar/baz/baz.proto
This results in two files having the path baz/baz.proto
. Imagine that a third file is thrown into the mix:
// THIS IS DEMONSTRATING SOMETHING BAD
syntax = "proto3";
package bar.baz;
import "baz/baz.proto";
Which file is being imported here? Is it foo/baz/baz.proto
? bar/baz/baz.proto
? The answer depends on the order of
the -I
flags given to protoc
, or (if buf
didn't error in this scenario pre-compilation, which buf
does) the
order of the imports given to the internal compiler.
While the above example is relatively contrived, the common error that comes up is when you have vendored .proto
files. For example, grpc-gateway has its own copy of the google.api definitions it needs.
While these are usually in sync, the google.api
schema can change. Imagine that we allowed this:
version: v2
# THIS IS INVALID AND RESULTS IN A PRE-COMPILATION ERROR
modules:
- path: proto
- path: vendor/github.com/googleapis/googleapis
- path: vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis
Which copy of google/api/*.proto
wins? The answer: no one wins. So Buf doesn't allow this.
2. Run build
You can run buf build
on your workspace by specifying the filepath to the directory containing the buf.yaml
configuration file. To target the module defined in the current directory:
$ buf build
The buf build
command:
- Discovers all Protobuf files per your
buf.yaml
configuration. - Copies the Protobuf files into memory.
- Compiles all Protobuf files.
- Outputs the compiled result to a configurable location (defaults to
/dev/null
)
If there are errors, they are printed out in a file:line:column:message
format by default. For example:
$ buf build
Outputacme/pet/v1/pet.proto:5:8:acme/payment/v1alpha1/payment.proto: does not exist
Build output can also be printed as JSON:
$ buf build --error-format=json
Output{"path":"acme/pet/v1/pet.proto","start_line":5,"start_column":8,"end_line":5,"end_column":8,"type":"COMPILE","message":"acme/payment/v1alpha1/payment.proto: does not exist"}
3. Options
Output format
By default, buf build
outputs its result to /dev/null
. In this case, it's common to use buf build
as a validation
step, analogous to checking if the input compiles.
buf build
also supports outputting FileDescriptorSet
s and images, which are Buf's
custom extension of the FileDescriptorSet
. Better yet, these outputs can be formatted in a variety of ways.
buf build
can deduce the output format by the file extension, see the documentation on automatically derived
formats. For example,
$ buf build -o image.binpb
$ buf build -o image.binpb.gz
$ buf build -o image.binpb.zst
$ buf build -o image.json
$ buf build -o image.json.gz
$ buf build -o image.json.zst
$ buf build -o image.txtpb
$ buf build -o image.txtpb.gz
$ buf build -o image.txtpb.zst
The special value -
is used to denote stdout, and you can manually set the format. For example:
$ buf build -o -#format=json
When combined with jq, buf build
also allows for introspection. For example, to see a list of all packages, you
can run this command:
$ buf build -o -#format=json | jq '.file[] | .package' | sort | uniq | head
Output"google.actions.type" "google.ads.admob.v1" "google.ads.googleads.v1.common" "google.ads.googleads.v1.enums" "google.ads.googleads.v1.errors" "google.ads.googleads.v1.resources" "google.ads.googleads.v1.services" "google.ads.googleads.v2.common" "google.ads.googleads.v2.enums" "google.ads.googleads.v2.errors"
Images always include the ImageExtension
field. But if you want a pure FileDescriptorSet
without this field set, and
thus to mimic protoc
entirely, you can use the --as-file-descriptor-set
flag:
$ buf build -o image.binpb --as-file-descriptor-set
The ImageExtension
field doesn't affect Protobuf plugins or any other operations, as they merely see this as an
unknown field. But we provide the option in case you want it.
Limit to specific files
By default, buf
builds all files under the buf.yaml
configuration file. You can instead manually specify the file or
directory paths to build. This is an advanced feature intended to be used for editor or Bazel integration—it's
better to let buf
discover all files under management and handle this for you.
The compiled result is limited to the given files if the --path
flag is specified, as in this command:
$ buf build --path path/to/foo.proto --path path/to/bar.proto
Limit to specific types
When you run buf build
to create a FileDescriptorSet
or Buf image, the output contains all
the Protobuf types declared in the module by default. But for some advanced use cases, you may want the image or
FileDescriptorSet
to contain only a subset of the types described in your Protobuf schemas.
Versions 1.1.0 and later of the buf
CLI include a --type
option for the buf build
command that enables you to
supply a fully qualified Protobuf name and limit the resulting image or FileDescriptorSet
to only those descriptors
required to represent those types and their required dependencies. This example usage restricts the output types to
those required to represent pkg.foo.Bar
:
$ buf build --type pkg.foo.Bar
The --type
flag accepts fully qualified names for messages, enums, and services. These dependent descriptors are
included in the build:
- Messages
- Messages and enums referenced in message fields
- Any proto2 extension declarations for message fields
- The parent message if this message is a nested definition
- Any custom options for the message, its fields, and the file in which the message is defined
- Enums
- The enum value descriptors for this enum
- The parent message is this enum is a nested definition
- Any custom options for the enum, enum values, and the file in which the enum is defined
- Services
- Request and response types referenced in service methods
- Any custom options for the services, its methods, and the file in which the service is defined
You can specify multiple types by applying the --type
option multiple times, as in this example:
$ buf build \
--type acme.weather.v1.Units \
--type acme.weather.v1.CurrentWeather.Temperature
In this case, dependent descriptors for both acme.weather.v1.Units
and acme.weather.v1.CurrentWeather.Temperature
are included in the output.
Type restriction example
As an example, consider these two .proto
files:
package pkg;
message Foo {
optional Bar bar = 1;
extensions 2 to 3;
}
message Bar {...}
message Baz {
other.Qux qux = 1 [(other.my_option).field = "buf"];
}
package other;
extend Foo {
optional Qux baz = 2;
}
message Qux{...}
message Quux{...}
extend google.protobuf.FieldOptions {
optional Quux my_option = 51234;
}
This table shows which files, messages, and extensions would be included for various types from foo.proto
and
bar.proto
if specified as the argument to --type
:
Type | Files | Messages | Extensions\ |
---|---|---|---|
buf build --type pkg.Foo | foo.proto , bar.proto | pkg.Foo , pkg.Bar , other.Qux | other.baz \ |
buf build --type pkg.Bar | foo.proto | pkg.Bar | |
buf build --type pkg.Baz | foo.proto , bar.proto | pkg.Baz , other.Quux , other.Qux | other.my_option |
5. Docker
Buf ships a Docker image bufbuild/buf that enables you to use buf
as part of your Docker workflow. For
example:
$ docker run \
--volume "$(pwd):/workspace" \
--workdir /workspace \
bufbuild/buf build