Remote plugin execution with the Buf Schema Registry
A while back, we announced that we had released the Buf Schema Registry (BSR), a centralized platform for managing Protobuf assets, into beta. In the announcement post, we briefly outlined a BSR feature called hosted plugins, which enables you to generate code using Protobuf plugins remotely, on the BSR, rather than on your local machine.
First, let's go over some of the basics of Protobuf plugins. A plugin is a program that a
Protobuf compiler, like the
buf CLI or protoc, executes to convert Protobuf sources
.proto files) into code stubs in your desired language, like Go, Java, C++, or Python. In order
to use plugins, a compiler needs to support Protobuf's plugin protocol, whereby:
- The compiler creates a
.protofiles passed into the compiler, such as
foo/bar.proto, and sends that request to the plugin on stdin.
- The Protobuf plugin accepts that request and writes a
- The compiler transforms that response into code files and writes them to disk.
Protobuf's original protoc compiler provides some built-in plugins but you need to install any
other plugins you need as executables in your environment. So if you needed to generate C++, Java,
Python, Go, and Rust code from a Protobuf source, for example, three of those plugins are built
into the compiler—C++, Java, and Python—but you'd need to install Go and Rust plugins on your own
protoc-gen-rust are widely used options).
With the Go and Rust plugins installed, you could generate code stubs using a command like this:
protoc \ -I proto \ --cpp_out=gen/proto/cpp \ --java_out=gen/proto/java \ --python_out=gen/proto/python \ --go_out=gen/proto/go \ --rust_out=gen/proto/rust \ proto/api.proto
version: v1 plugins: - name: cpp out: gen/proto/cpp - name: java out: gen/proto/java - name: python out: gen/proto/python - name: go out: gen/proto/go - name: rust out: gen/proto/rust
...and generate your code stubs with a single command:
buf generate <input>
<input> can be any valid Buf input, such as a directory, tarball, or Buf module.
This is vastly more flexible than protoc, which only supports
.proto files as sources.
Problems with local plugin execution
As you can see, the
buf CLI is a major improvement over protoc, which you no longer need to
install or manage. But there is one remaining problem here: to run this example, you'd still need
to install plugins locally and ensure that you're using the right plugin versions.
One half-solution is to either manage those executables yourself or rely on custom tooling, which frequently leads to those classic "works on my machine" reproducibility problems where you need to hunt down subtle differences in plugin versions used in different environments.
The difficulty of managing this complex web of compiler and plugin versions—just to make code stub generation work—has been a major barrier to Protobuf adoption. At Buf, we decided to fix this once and for all by removing the need for local plugins entirely.
Remote plugin execution
With remote plugin execution, you can publish Protobuf plugins to the BSR and then execute those plugins on a trusted server—the BSR itself—rather than on your local machine.
Authoring a plugin involves two steps: building the plugin as a Docker image and pushing
the image to
plugins.buf.build. Once your plugin has been pushed, you need to make one change to
buf.gen.yaml configuration file to use it:
version: v1 plugins: - remote: buf.build/my-buf-org/plugins/my-proto-plugin:v1.2.3 out: gen/proto/my-language
Note the usage of the
remote key rather than
name. With this configuration, the
command now makes a remote procedure call (RPC) to the BSR to execute the
remote plugin on your
behalf. Here's an example generation command:
buf generate buf.build/acme/petapis
buf CLI would generate code for the
buf.build/acme/petapis module using the local
buf.gen.yaml configuration. But instead of calling a local
executable, all code generation would happen on the BSR, and the resulting files would be written
This diagram illustrates that process:
Why it matters
The Buf team has developed processes to automatically sync and publish all of protoc's built-in
plugins to the BSR under the
buf.build/protocolbuffers organization. This means that you
can remove any and all plugin executables from your environment—local, CI/CD, whatever—and
rely solely on remote plugins.
These protoc built-in plugins are hosted on the BSR:
We also host many popular gRPC plugins, including:
Beyond that, some community members have made their own plugins publicly available on the BSR. We expect the number of community plugins to steadily expand over time.
You can replace protoc with the
buf CLI regardless of how you're using plugins. But using local
buf does have the drawback that you still need to install and manage those plugins
yourself. Remote plugin execution goes one step further and enables you to eliminate not just protoc
but even local plugin executables from your Protobuf workflows, making it easier than ever to
introduce Protobuf into your environments.
Later in this series, we'll highlight some other BSR features, such as generated documentation for Protobuf and remote code generation.