Skip to content

Code generation – Overview#

This overview explains how to generate code from your Protobuf files using the Buf CLI, ranging from local generation to generated SDKs hosted in the Buf Schema Registry (BSR).

One of the challenges with Protobuf code generation is the complexity of working with protoc and plugins. Managing and maintaining a stable environment locally on a single machine is hard enough given the complex web of different compiler and plugin versions. The problem is compounded as you scale out code generation across many developers, and often results in a series of ugly bash scripts shared between team members.

Buf's code generation streamlines all of that and is designed to work with whatever your Protobuf setup might be. It has the following advantages over protoc:

  • Speed: Compiles 2x faster than protoc.
  • Clarity: Allows you to configure your build once and easily edit or reference as needed from source control, instead of memorizing strings of flags.
  • Flexibility: Accepts more types of input (protoc only accepts raw .proto files) and the input doesn't have to be local.

In addition, Buf offers several features that ease many of the frustrations of generating client code from Protobuf files, especially in larger organizations:

  • Remote plugins: Avoid installing and maintaining protoc plugins locally. Specify our hosted versions instead, right down to the version number, so it's easy to standardize across teams or use different versions for different projects as needed.
  • Managed mode: Produce universal APIs by removing user- and language-specific Protobuf options from .proto files. API consumers can then enable smart defaults for these options in the config file with two lines of code and never think about them again.
  • Generated SDKs: Skip the entire code generation process and consume your APIs via NPM, Maven, and other common dependency management tools. They're created automatically when modules are pushed to the BSR.

The sections below discuss key concepts, local generation (with and without remote plugins), managed mode, and specific invocations of the buf generate command.

See the quickstart for a step-by-step walkthrough of setting up code generation, basic usage, and using remote plugins and managed mode.

Key concepts#

protoc
protoc is the compiler included with Protocol Buffers—it invokes a set of plugins to generate code or perform other custom tasks.
input
To the Buf CLI, an input is the representation of the Protobuf files you want to manage. It can take many forms, such as a local directory of .proto files, a module hosted in the Buf Schema Registry, or a .zip file. See the Inputs reference for a full list of inputs that the Buf CLI accepts.
plugins
Plugins are external programs that implement a specific Protobuf interface. They're used by protoc and Buf to generate code or perform other custom processing on Protobuf files. With the Buf CLI, you can either install and use plugins locally, or save time and improve consistency by referencing the remote plugins that we host in the Buf Schema Registry (BSR). They're the same as the ones you locally install.
buf.gen.yaml
This is the Buf config file specifically related to code generation. It's usually placed at the root of your workspace and specifies the plugin to be used for each output, the output path, and set of options that enables you to customize the generated code further.

To see all of the available configuration options, go to the buf.gen.yaml reference.

Generating with local plugins#

Generating client code with locally installed plugins works similarly to protoc, but faster. For plugins that are built into protoc, use the plugins.protoc_builtin key and specify the language as the value:

buf.gen.yaml – Generate code using the built-in protoc C++ plugin
version: v2
plugins:
  - protoc_builtin: cpp
    out: gen/cpp

Generating with remote plugins#

Managing a set of local plugins can complicate your workflow, especially if you're trying to be consistent across teams. A common solution is to rely on custom tooling, which frequently leads to "works on my machine" reproducibility problems. With Buf, you can standardize code generation by specifying the mirrors of the protoc plugins that we host in the BSR. Just point the plugins.remote key to the BSR, and it generates the code.

buf.gen.yaml – Generate code using a remote plugin
version: v2
plugins:
  - remote: buf.build/protocolbuffers/go
    out: gen/go

For more information, see the remote plugins overview.

Using managed mode#

Another issue with code generation is the need to define output options that aren't part of the schema. These include fields like package prefixes, namespace information, and so on that aren't part of your API definition but need to be passed to the generator to render the client code correctly. Using managed mode allows consumers to easily use smart defaults for these options or override them in the buf.gen.yaml file, freeing producers to provide a universal API.

For more information, see the Managed mode page.

Next steps#

  • Quickstart: Step-by-step walkthrough of setting up code generation
  • Usage guide: Command-line examples and advanced usage patterns
  • Managed mode: Remove language-specific options from your .proto files
  • Troubleshooting: Common issues and solutions