Tutorial - Remote plugins
We highly recommend completing the tour to get an overview of
buf generate
with remote plugins.
Protobuf code generation is a challenging process for developers due to the complexities of working with protoc and plugins. This challenge is compounded as code generation is scaled across multiple developers, with different languages and runtime requirements for plugins.
To overcome these challenges, organizations and open-source projects often develop their own homegrown Protobuf tooling to simplify the developer experience and maintain consistent output. However, these solutions are often difficult to maintain over time, and knowledge of the solutions can be lost as staff leave the organization or project.
To provide a more streamlined approach to code generation, Buf's Remote Plugins remove the need for developers to manage, download, or run plugins on their local machines. This approach ensures a consistent environment and provides a necessary solution for the Protobuf ecosystem. Let's take a look at how you can simplify your code generation workflow with generated SDKs:
Configuration
The buf.gen.yaml
file controls how the buf generate
command executes Protobuf plugins for any
input. Here, we can specify remote plugins to perform code generation. For
more information on the buf.gen.yaml
configuration, see the reference.
Buf verifies and maintains the commonly-used plugins used across the Protobuf ecosystem. To discover all publicly available plugins, go to buf.build/plugins.
1. Choose your input
Remote plugins generate code for inputs. An input can be a git repository, tarball, zip file,
a or local directory containing Protobuf files configured with a buf.yaml
configuration file. Buf refers to such directories of Protobuf files as modules.
For our purposes, we'll assume you have a buf.yaml
in your current directory that defines a module. To create a
buf.yaml
in your current directory.
$ buf mod init
version: v1
breaking:
use:
- FILE
lint:
use:
- DEFAULT
See the build documentation for full details on how to configure modules to build with buf
.
2. Create a buf.gen.yaml
Now that you have an input to generate code for, we need to define a
buf.gen.yaml
and specify what protoc
plugins you want to use.
A remote plugin can be pinned to a specific version, or if you want to always use the latest version of the plugin, the version can be omitted. Here's some examples for common languages and RPC frameworks:
version: v1
plugins:
- plugin: buf.build/protocolbuffers/go
out: gen/go
opt: paths=source_relative
- plugin: buf.build/grpc/go
out: gen/go
opt: paths=source_relative
version: v1
plugins:
- plugin: buf.build/bufbuild/es
out: gen/js
- plugin: buf.build/bufbuild/connect-es
out: gen/js
version: v1
plugins:
- plugin: buf.build/protocolbuffers/python
out: gen/python
- plugin: buf.build/grpc/python
out: gen/python
version: v1
plugins:
- plugin: buf.build/protocolbuffers/ruby
out: gen/ruby
- plugin: buf.build/grpc/ruby
out: gen/ruby
version: v1
plugins:
- plugin: buf.build/protocolbuffers/java
out: gen/java
- plugin: buf.build/grpc/java
out: gen/java
version: v1
plugins:
- plugin: buf.build/apple/swift
opt: Visibility=Public
out: gen/swift
- plugin: buf.build/connectrpc/swift
opt: Visibility=Public
out: gen/swift
Note, we're using the plugin
key to reference a remote plugin. More information can be
found in the buf.gen.yaml docs.
Plugins are invoked in the order they are specified in your buf.gen.yaml
with results from each invocation combined
before writing the result. For more information, see the
buf.gen.yaml
reference.
As a best practice, when referencing remote, plugins we recommend including the version of the plugin to ensure reproducible code generation. It is possible to reference both local and remote plugins within a single template file.
3. Run generate
To generate using the module and buf.gen.yaml
defined in your current directory:
$ buf generate
The buf generate
command will:
- Send the Protobuf files specified in your input to the Buf Schema Registry remote plugin executor.
- Invoke each plugin specified in your
buf.gen.yaml
. - Send the results back, and unpack them on your local file system.
By default, buf generate
looks for a buf.gen.yaml
in your current
directory. An alternate location can be specified by using the --template
flag:
$ buf generate --template templates/buf.go.gen.yaml
You can also run buf generate
. For example if all of your .proto
files are in directory foo
:
# generate using the module defined in the foo/ directory
$ buf generate foo
# generate using the files contained in the root directory
# of the github.com/acme/weather repository on the default branch
$ buf generate https://github.com/acme/weather.git
# generate using the buf.gen.yaml file at templates/buf.go.gen.yaml
# using the files in the buf.build/grpc/grpc module
$ buf generate --template templates/buf.go.gen.yaml buf.build/acme/petapis
If you used one of the example buf.gen.yaml
files from above, you should end up with this file structure:
.
├── buf.gen.yaml
└── gen
└── go
└── pet
└── v1
├── pet.pb.go
└── pet_grpc.pb.go
.
├── buf.gen.yaml
└── gen
└── js
└── pet
└── v1
├── pet_connect.d.ts
├── pet_connect.js
├── pet_pb.d.ts
└── pet_pb.js
.
├── buf.gen.yaml
└── gen
└── python
└── pet
└── v1
├── pet_pb2.py
└── pet_pb2_grpc.py
.
├── buf.gen.yaml
└── gen
└── ruby
└── pet
└── v1
├── pet_pb.rb
└── pet_services_pb.rb
.
├── buf.gen.yaml
└── gen
└── java
└── pet
└── v1
├── PetOuterClass.java
└── PetStoreServiceGrpc.java
.
├── buf.gen.yaml
└── gen
└── swift
└── pet
└── v1
├── pet.connect.swift
└── pet.pb.swift
4. Common use cases
4.1. protoc-gen-go
protoc-gen-go
generates message and enum stubs for Go. The BSR hosts
this plugin at buf.build/protocolbuffers/go.
version: v1
plugins:
- plugin: buf.build/protocolbuffers/go
out: gen/go
# You almost always want to set this option with protoc-gen-go
opt: paths=source_relative
4.2. protoc-gen-go + protoc-gen-validate
protoc-gen-validate
is a plugin that generates polyglot message
validators. The BSR hosts this plugin pre-enabled for Go code generation at
buf.build/bufbuild/validate-go.
version: v1
plugins:
# The protoc-gen-go stubs are required for grpc-go
- plugin: buf.build/protocolbuffers/go
out: gen/go
# You almost always want to set this option with protoc-gen-go
opt: paths=source_relative
- plugin: buf.build/bufbuild/validate-go
# Make sure to generate your protoc-gen-validate code to the same
# directory as protoc-gen-go
out: gen/go
# You almost always want to set this option with protoc-gen-go
opt: paths=source_relative
4.3. grpc-go
protoc-gen-go-grpc
generates Go service stubs
for gRPC. The BSR hosts this plugin at buf.build/grpc/go. Note that we'd recommend checking
out connect-go instead of using grpc-go (see the next section).
version: v1
plugins:
# The protoc-gen-go stubs are required for grpc-go
- plugin: buf.build/protocolbuffers/go
out: gen/go
# You almost always want to set this option with protoc-gen-go
opt: paths=source_relative
- plugin: buf.build/grpc/go
# Make sure to generate your grpc-go code to the same
# directory as protoc-gen-go
out: gen/go
# You almost always want to set this option with protoc-gen-go-grpc
opt: paths=source_relative
4.4. connect-go
Connect-Go is a slim library for building browser and gRPC-compatible HTTP APIs. Handlers and clients support three protocols: gRPC, gRPC-Web, and Connect's own protocol.
protoc-gen-connect-go
generates Go service stubs for Connect. The BSR
hosts this plugin at buf.build/connectrpc/go.
version: v1
plugins:
# The protoc-gen-go stubs are required for grpc-go
- plugin: buf.build/protocolbuffers/go
out: gen/go
# You almost always want to set this option with protoc-gen-go
opt: paths=source_relative
- plugin: buf.build/connectrpc/go
# Unlike grpc-go, connect stubs do not need to be generated to the
# same directory, however you are free to do so
out: gen/go
# You almost always want to set this option with protoc-gen-connect-go
opt: paths=source_relative
4.5. connect-es
Connect-ES brings the Connect ecosystem to TypeScript, the web browser, and to Node.js. It contains packages for working with Connect and gRPC-Web clients from the browser as well as Connect-, gRPC-, and gRPC-Web-compatible clients and servers in Node.js
protoc-gen-connect-es
generates client service
stubs for TypeScript and JavaScript. It requires the
protoc-gen-es
plugin to generate message and service stubs.
The code generated by these two plugins requires the runtime libraries
@connectrpc/connect
, and
@bufbuild/protobuf
.
The BSR hosts these plugins at buf.build/connectrpc/es and buf.build/bufbuild/es.
version: v1
plugins:
- plugin: buf.build/bufbuild/es
out: gen/es
- plugin: buf.build/connectrpc/es
out: gen/es
4.6. connect-swift
Connect-Swift is a small library that provides support for using generated, type-safe, and idiomatic Swift APIs to communicate with your app's servers. It can be used with both the gRPC-Web and Connect protocols.
protoc-gen-connect-swift
is responsible for generating Swift clients,
and relies on the models generated by protoc-gen-swift
. The BSR hosts both
of these plugins at buf.build/connectrpc/connect-swift and
buf.build/apple/swift, respectively.
To get started with Connect-Swift, check out the demo tutorial.
version: v1
plugins:
- plugin: buf.build/apple/swift
opt: Visibility=Public
out: gen/swift
- plugin: buf.build/connectrpc/swift
opt: Visibility=Public
out: gen/swift
Related docs
If you're interested in learning more about code generation with Protobuf, or how Buf can help streamline your development process, be sure to check out the generated SDKs tutorial