Generating code with remote plugins – Tutorial

We highly recommend completing the tour to get an overview of buf generate with remote plugins.

Protobuf code generation is a challenging process for developers due to the complexities of working with protoc and plugins. This challenge is compounded as code generation is scaled across multiple developers, with different languages and runtime requirements for plugins.

To overcome these challenges, organizations and open-source projects often develop their own homegrown Protobuf tooling to simplify the developer experience and maintain consistent output. However, these solutions are often difficult to maintain over time, and knowledge of the solutions can be lost as staff leave the organization or project.

To provide a more streamlined approach to code generation, Buf's Remote Plugins remove the need for developers to manage, download, or run plugins on their local machines. This approach ensures a consistent environment and provides a necessary solution for the Protobuf ecosystem. Let's take a look at how you can simplify your code generation workflow with generated SDKs:

Configuration

The buf.gen.yaml file controls how the buf generate command executes Protobuf plugins for any input. Here, you can specify remote plugins to perform code generation. For more information on the buf.gen.yaml configuration, see the reference.

Buf verifies and maintains the commonly-used plugins used across the Protobuf ecosystem. To discover all publicly available plugins, go to buf.build/plugins.

1. Choose your input

Remote plugins generate code for inputs. An input can be a Git repository, tarball, zip file, or a local directory containing Protobuf files configured with a buf.yaml configuration file. Buf refers to such directories of Protobuf files as modules.

For our purposes, we'll assume you have a directory of .proto files with a buf.yaml configuration file that defines them as a workspace with at least one module. To create a buf.yaml in your current directory if you don't have one, run this command:

$ buf config init

It creates a buf.yaml with the default settings:

version: v2
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

Add a modules declaration with the path to the directory that contains your .proto files.

version: v2
modules:
  - path: /path/to/proto/files
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

Your directory structure should look like this:

workspace_root
├── buf.yaml
└── proto_file_directory

2. Create a buf.gen.yaml file

Now that you have an input to generate code for, you need to define a buf.gen.yaml file and specify which protoc plugins you want to use. Copy the code for the language or framework you want to generate for and create a new buf.gen.yaml in the same directory as the buf.yaml file:

version: v2
plugins:
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    opt: paths=source_relative
  - remote: buf.build/grpc/go
    out: gen/go
    opt: paths=source_relative

version: v2
plugins:
  - remote: buf.build/bufbuild/es
    out: gen/js
  - remote: buf.build/bufbuild/connect-es
    out: gen/js

version: v2
plugins:
  - remote: buf.build/protocolbuffers/python
    out: gen/python
  - remote: buf.build/grpc/python
    out: gen/python

version: v2
plugins:
  - remote: buf.build/protocolbuffers/ruby
    out: gen/ruby
  - remote: buf.build/grpc/ruby
    out: gen/ruby

version: v2
plugins:
  - remote: buf.build/protocolbuffers/java
    out: gen/java
  - remote: buf.build/grpc/java
    out: gen/java

version: v2
plugins:
  - remote: buf.build/apple/swift
    opt: Visibility=Public
    out: gen/swift
  - remote: buf.build/connectrpc/swift
    opt: Visibility=Public
    out: gen/swift

Note that the code uses the remote key to reference a remote plugin. For more details about the buf.gen.yaml fields, see the configuration file docs.

Plugins are invoked in the order they are specified in buf.gen.yaml, with results from each invocation combined before writing the result.

When referencing remote plugins, we recommend including the version of the plugin to ensure reproducible code generation. It's possible to reference both local and remote plugins within a single buf.gen.yaml file.

3. Run generate

To generate using the module and buf.gen.yaml you defined, run this command from the root of the workspace:

$ buf generate

The buf generate command will:

• Send the Protobuf files specified in your input to the Buf Schema Registry remote plugin executor.
• Invoke each plugin specified in your buf.gen.yaml.
• Send the results back, and unpack them on your local file system.

By default, buf generate looks for a buf.gen.yaml in your current directory. An alternate location can be specified by using the --template flag:

$ buf generate --template templates/buf.go.gen.yaml

If you used one of the example buf.gen.yaml files from above, you should end up with this file structure:

workspace_root
├── buf.gen.yaml
└── gen
    └── go
        └── pet
            └── v1
                ├── pet.pb.go
                └── pet_grpc.pb.go

workspace_root
├── buf.gen.yaml
└── gen
    └── js
        └── pet
            └── v1
                ├── pet_connect.d.ts
                ├── pet_connect.js
                ├── pet_pb.d.ts
                └── pet_pb.js

workspace_root
├── buf.gen.yaml
└── gen
    └── python
        └── pet
            └── v1
                ├── pet_pb2.py
                └── pet_pb2_grpc.py

workspace_root
├── buf.gen.yaml
└── gen
    └── ruby
        └── pet
            └── v1
                ├── pet_pb.rb
                └── pet_services_pb.rb

workspace_root
├── buf.gen.yaml
└── gen
    └── java
        └── pet
            └── v1
                ├── PetOuterClass.java
                └── PetStoreServiceGrpc.java

workspace_root
├── buf.gen.yaml
└── gen
    └── swift
        └── pet
            └── v1
                ├── pet.connect.swift
                └── pet.pb.swift

4. Common use cases

4.1. protoc-gen-go

protoc-gen-go generates message and enum stubs for Go. The BSR hosts this plugin at buf.build/protocolbuffers/go.

version: v2
plugins:
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    # You almost always want to set this option with protoc-gen-go
    opt: paths=source_relative

4.2. protoc-gen-go + protoc-gen-validate

protoc-gen-validate is a plugin that generates polyglot message validators. The BSR hosts this plugin pre-enabled for Go code generation at buf.build/bufbuild/validate-go.

version: v2
plugins:
  # The protoc-gen-go stubs are required for grpc-go
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    # You almost always want to set this option with protoc-gen-go
    opt: paths=source_relative
  - remote: buf.build/bufbuild/validate-go
    # Make sure to generate your protoc-gen-validate code to the same
    # directory as protoc-gen-go
    out: gen/go
    # You almost always want to set this option with protoc-gen-go
    opt: paths=source_relative

4.3. grpc-go

protoc-gen-go-grpc generates Go service stubs for gRPC. The BSR hosts this plugin at buf.build/grpc/go. Note that we'd recommend checking out connect-go instead of using grpc-go (see the next section).

version: v2
plugins:
  # The protoc-gen-go stubs are required for grpc-go
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    # You almost always want to set this option with protoc-gen-go
    opt: paths=source_relative
  - remote: buf.build/grpc/go
    # Make sure to generate your grpc-go code to the same
    # directory as protoc-gen-go
    out: gen/go
    # You almost always want to set this option with protoc-gen-go-grpc
    opt: paths=source_relative

4.4. connect-go

Connect-Go is a slim library for building browser and gRPC-compatible HTTP APIs. Handlers and clients support three protocols: gRPC, gRPC-Web, and Connect's own protocol.

protoc-gen-connect-go generates Go service stubs for Connect. The BSR hosts this plugin at buf.build/connectrpc/go.

version: v2
plugins:
  # The protoc-gen-go stubs are required for grpc-go
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    # You almost always want to set this option with protoc-gen-go
    opt: paths=source_relative
  - remote: buf.build/connectrpc/go
    # Unlike grpc-go, connect stubs do not need to be generated to the
    # same directory, however you are free to do so
    out: gen/go
    # You almost always want to set this option with protoc-gen-connect-go
    opt: paths=source_relative

4.5. connect-es

Connect-ES brings the Connect ecosystem to TypeScript, the web browser, and to Node.js. It contains packages for working with Connect and gRPC-Web clients from the browser as well as Connect-, gRPC-, and gRPC-Web-compatible clients and servers in Node.js

protoc-gen-connect-es generates client service stubs for TypeScript and JavaScript. It requires the protoc-gen-es plugin to generate message and service stubs. The code generated by these two plugins requires the runtime libraries @connectrpc/connect, and @bufbuild/protobuf.

The BSR hosts these plugins at buf.build/connectrpc/es and buf.build/bufbuild/es.

version: v2
plugins:
  - remote: buf.build/bufbuild/es
    out: gen/es
  - remote: buf.build/connectrpc/es
    out: gen/es

4.6. connect-swift

Connect-Swift is a small library that provides support for using generated, type-safe, and idiomatic Swift APIs to communicate with your app's servers. It can be used with both the gRPC-Web and Connect protocols.

protoc-gen-connect-swift is responsible for generating Swift clients, and relies on the models generated by protoc-gen-swift. The BSR hosts both of these plugins at buf.build/connectrpc/connect-swift and buf.build/apple/swift, respectively.

To get started with Connect-Swift, check out the demo tutorial.

version: v2
plugins:
  - remote: buf.build/apple/swift
    opt: Visibility=Public
    out: gen/swift
  - remote: buf.build/connectrpc/swift
    opt: Visibility=Public
    out: gen/swift

Related docs

If you're interested in learning more about code generation with Protobuf, or how Buf can help streamline your development process, be sure to check out the generated SDKs tutorial