Code generation overview
This overview provides information about generating code from your Protobuf files using the Buf CLI, ranging from local generation to generated SDKs hosted in the Buf Schema Registry (BSR).
One of the challenges with Protobuf code generation is the complexity of working with protoc
and plugins. Managing and
maintaining a stable environment locally on a single machine is hard enough given the complex web of different compiler
and plugin versions. The problem is compounded as you scale out code generation across many developers, and often
results in a series of ugly bash scripts shared between team members.
Buf's code generation streamlines all of that and is designed to work with whatever your Protobuf setup might be. It has
the following advantages over protoc
:
- Speed: Compiles 2x faster than
protoc
. - Clarity: Allows you to configure your build once and easily edit or reference as needed from source control, instead of memorizing strings of flags.
- Flexibility: Accepts more types of input (
protoc
only accepts raw.proto
files) and the input doesn't have to be local.
In addition, Buf offers several features that ease many of the frustrations of generating client code from Protobuf files, especially in larger organizations:
- Remote plugins: Avoid installing and maintaining
protoc
plugins locally. Specify our hosted versions instead, right down to the version number, making it easy to standardize across teams or use different versions for different projects, as needed. - Managed mode: Produce universal APIs by removing user and language specific Protobuf options from
.proto
files. API consumers can then enable smart defaults for these options in the config file with two lines of code and never think about them again. - Generated SDKs: Skip the entire code generation process and use generated SDKs. The BSR packages up your modules so they can be consumed via NPM, Maven, and other common dependency management tools. They're created automatically when modules are pushed to the BSR.
The sections below discuss key concepts, local generation (with and without remote plugins), managed mode, and specific
invocations of the buf generate
command.
See the tutorial for a step-by-step walkthrough of setting up code generation, basic usage, and using remote plugins and managed mode.
Key concepts
protoc
protoc
is the compiler included with Protocol Buffers—it invokes a set of plugins to generate code or perform other custom tasks.- input
- To the Buf CLI, an input is the representation of the Protobuf files you want to manage. It can take many forms,
such as a local directory of
.proto
files, a module hosted in the Buf Schema Registry, or a.zip
file. See the Inputs reference for a full list of inputs that the Buf CLI accepts. - plugins
- Plugins are external programs that implement a specific Protobuf interface. They're used by
protoc
and Buf to generate code or perform other custom processing on Protobuf files. With the Buf CLI, you can either install and use plugins locally, or save time and improve consistency by referencing the remote plugins that we host in the Buf Schema Registry (BSR). They are the same as the ones you locally install. buf.gen.yaml
- This is the Buf config file specifically related to code generation. It's generally placed at the root of your workspace and specifies the plugin to be used for each output, the output path, a set of options that enable you to customize the generated code further.
To see all of the available configuration options, go to the
buf.gen.yaml
reference.
Generating with local plugins
Generating client code with locally-installed plugins works basically the same as protoc
, but faster. In the diagram
below, the plugins.local
key specifies a local version of protoc-gen-go
.
For plugins that are built into protoc
, use the plugins.protoc_builtin
key and specify the language as the value:
version: v2
plugins:
- protoc_builtin: cpp
out: gen/cpp
The protoc_builtin
key accepts these values: cpp
, csharp
, java
, objectivec
, php
, python
, or ruby
.
Otherwise, use the plugins.local
key and provide the name of a plugin on your PATH
:
version: v2
plugins:
- local: protoc-gen-go
out: gen/go
Generating with remote plugins
As noted above, managing a set of local plugins can complicate your workflow, especially if you're trying to be
consistent across teams. A common solution is to rely on custom tooling, which frequently leads to "works on my machine"
reproducibility problems. With Buf, you can standardize code generation by specifying the mirrors of the protoc
plugins that we host in the BSR. Just point the plugins.remote
key to the BSR, and it generates the code.
version: v2
plugins:
- remote: buf.build/protocolbuffers/go
out: gen/go
For more information, see Using remote plugins.
Using managed mode
Another issue with code generation is the need define output options that aren't part of the schema. These include fields like package prefixes, namespace information, and so on that aren't part of your API definition but need to be passed to the generator to render the client code correctly.
Normally, API producers hard-code these options in their Protobuf files, but they are really API consumer concerns.
Using managed mode allows consumers to easily use smart defaults for these options or override them in the
buf.gen.yaml
file, freeing the producers to provide a universal API. Using managed mode across an entire project or
organization gives you a clear separation of concerns while still providing necessary flexibility to consumers.
The example below demonstrates using managed mode to generate Java code using remote plugins.
Before
syntax = "proto3";
package acme.weather.v1;
option java_multiple_files = true;
option java_outer_classname = "WeatherProto";
option java_package = “com.acme.weather.v1”;
version: v2
plugins:
- remote: buf.build/protocolbuffers/java
out: gen/proto/java
After
syntax = "proto3";
package acme.weather.v1;
version: v2
managed:
enabled: true
plugins:
- remote: buf.build/protocolbuffers/java
out: gen/proto/java
For more information, see the Managed mode and
buf.gen.yaml
reference pages.
Command-line input examples
The examples below show ways of invoking buf generate
for different inputs or subsets of inputs. Generally, you
should specify all inputs in the buf.gen.yaml
file, but they can also be passed to buf generate
at the command line
as in the examples below. If you specify an input in buf.gen.yaml
and in the command, the command takes precedence
and only that input is generated.
See the buf.gen.yaml
configuration file documentation to learn how to specify
inputs there.
$ buf generate
$ buf generate buf.build/acme/petapis
$ buf generate https://github.com/foo/bar.git
Limit to specific files
By default, the Buf CLI builds all files in the workspace. You can instead manually specify the file or directory paths to build. This is an advanced feature intended to be used for editor or Bazel integration—it's better to let the Buf CLI discover all files under management and handle this for you.
If you only want to generate code for a subset of your input, you can do so via the --path
flag:
Only generate for the files in the directories proto/foo
and proto/bar
$ buf generate --path proto/foo --path proto/bar
Only generate for the files proto/foo/foo.proto
and proto/foo/bar.proto
$ buf generate --path proto/foo/foo.proto --path proto/foo/bar.proto
Only generate for the files in the directory proto/foo
in your GitHub repository
$ buf generate https://github.com/foo/bar.git --template data/generate.yaml --path proto/foo
Limit to specific types
By default, buf
generates code for all types in your source file(s). However, you can request that it only
generate the types that you absolutely need by specifying a list of types you want to include in the buf.gen.yaml
,
omitting everything else from the generated output. The resulting schema contains these elements and all of their
dependencies, according to the following principles. This is especially useful for mobile and web apps, where shipping
unused code is expensive.
- All names must be fully-qualified. If any name is unknown, the request will fail and no output will be returned.
- When a message is included, any type references therein, such as from fields that refer to other message or enum
types, are also included. If any included message is extendable (a
proto2
-only feature), its known extensions will be included. - If a particular method is included, its request and response message types will also be included, but other methods that belong to the same service will not (unless they are also included).
- If a service is included, all of its methods (and their request and response types) are included.
- If any included types have custom options, the definitions for those custom options will also be included.
If you only want to generate code for a subset of your types, you can specify it like this:
version: v1
types:
include:
- buf.alpha.lint.v1.IDPaths
plugins:
- local: protoc-gen-go
out: gen/go
opt: paths=source_relative
- local: go-grpc
out: gen/go
opt:
- paths=source_relative
- require_unimplemented_servers=false
You can also use the --type
flag, which overrides whatever might be configured in buf.gen.yaml
.
Only generate the type with a fully qualified name of buf.alpha.lint.v1.IDPaths
$ buf generate --type buf.alpha.lint.v1.IDPaths
Generate to a specific directory
You can generate to a specific output directory using the --output
or -o
flag. This command generates to the bar/
directory while prepending bar/
to the out
directives in the buf.gen.yaml
file:
$ buf generate https://github.com/foo/bar.git -o bar
The paths in the config file and the -o
flag are interpreted as relative to your current directory, so you can
place your config files anywhere.
Generate with a configuration file in another directory
The config file doesn't have to be named buf.gen.yaml
, nor does it have to be in the directory where you're invoking
buf generate
. To specify a config file elsewhere, use the --template
flag:
$ buf generate buf.build/acme/petapis --template templates/generation.yaml
Generate using multiple configuration files
For cases where you need to use different output configurations for the same input, we recommend using multiple configuration files with different names.
To use one file for Go and a different file for Java, for example, you could create a buf.gen.go.yaml
file and a
buf.gen.java.yaml
file and use separate commands to generate code:
$ buf generate --template buf.gen.go.yaml
$ buf generate --template buf.gen.java.yaml
Generate without a configuration file
Although we recommend using a configuration file, without one buf generate
will assume the input is the current
directory. You can also specify the config file as JSON in the invocation. This
can be useful when configuring CI/CD builds for an organization when you want to centralize the configuration for all
teams.
$ buf generate --template '{"version":"v2","plugins":[{"protoc-builtin":"go","out":"gen/go"}]}'
Related docs
- Try out code generation with the
buf generate
tutorial - Learn more about consuming generated SDKs instead of generating code
- Browse the
buf generate
command reference - Find out how to create custom plugins for your organization