Modules and workspaces

All Buf operations and features work with collections of Protobuf files that you configure, rather than specifying file paths on the command line. This page describes how to define these collections for both your local environment and for pushing to the Buf Schema Registry (BSR) to share with your team, customers, or the wider Buf community.

A module is the key primitive in the Buf ecosystem, and represents a collection of Protobuf files that are configured, built, and versioned as a logical unit. Modules simplify file discovery and eliminate the need for complex protoc build scripts that define your Protobuf sources with -I. Instead, you configure a module in the buf.yaml file, which specifies the location of the files, any exclusions, and its linting and breaking change detection rule sets if they're different from the workspace defaults.

Workspaces are collections of modules that are defined together in the same buf.yaml file to create a local working environment—they can contain a single module or multiple modules. All modules in a workspace can import each other without specific dependency declarations, and also share any dependencies that are defined for the workspace (for example, on an external module such as googleapis).

Workspaces are also the default input for Buf CLI commands, and are the unit used to push content to the Buf Schema Registry (BSR). Pushing the workspace means that all modules in the workspace will be pushed to their respective BSR repositories.

Workspace layout

Though each module in the workspace is already a versioned entity composed of Protobuf files, we recommend implementing some level of versioning in the workspace's directory and package structure as well, even if it only contains a single module. Structuring the files this way decreases the likelihood of API collisions with other user-defined schemas. For more details, see Files and packages.

For example, suppose you want to create a workspace with two modules, vendor/units/v1 and acme/weatherapi/v1. It would ideally have a directory structure that mirrors the packages, with the .proto files nested by package in a separate folder. All Buf config files should be at the root of the workspace.

workspace_root
├─ buf.gen.yaml
├─ buf.lock
├─ buf.yaml
├─ proto
│  └─ acme
│     └─ weatherapi
│        └─ v1
│           ├── api.proto
│           ├── calculate.proto
│           └── conversion.proto
├─ vendor
│  └─ units
│     └─ v1
│        ├─ imperial.proto
│        └─ metric.proto
├─ LICENSE
└─ README.md

Module documentation

In addition to comments associated with your Protobuf definitions, we recommend adding a README.md file at the module root to describe the module's overall function. The module README.md file is similar to a GitHub repository's README.md and currently supports all CommonMark syntax.

Because we consider this an integral part of the module, any changes made to the README.md file trigger a new commit in the BSR.

Module license

Public repositories on the public BSR are often used to share open source software. For your repository to truly be open source, you need to license it so that others are free to use, change, and distribute the software.

As a best practice, we encourage you to include the license file with your module. To do this, simply include a LICENSE file at the module root and push it to the BSR.

Disclaimer

We aim to provide users with open source license information to help them make informed decisions. However, we are not legal experts and do not guarantee the accuracy of the information provided. We recommend consulting with a professional for any legal issues related to open source licenses.

Workspace and module configuration

You define workspaces and modules by creating a buf.yaml configuration file at the root directory of the workspace (generally, this should be the root of your source control repository). It specifies the workspace properties and those of the modules it contains. All module paths must be contained within the workspace directory. Given the directory shown above, the buf.yaml file would look like this:

# Buf configuration version.
version: v2

# A list of modules included in the local workspace. Each module
# defines the path to the directory where the Protobuf files are, and optionally
# its name (which corresponds a repository on the BSR), files or directories to
# exclude, and module-level lint and breaking change settings.
#
# Both the path and any excludes fields are relative to the root of the workspace.
modules:
  - path: proto
    name: buf.build/acme/weatherapi
  - path: vendor
    # Lint and breaking change detection settings within a module completely
    # replace the workspace-level defaults for that module. There's no rule
    # merging or de-duping.
    lint:
      use:
        - MINIMAL
    breaking:
      use:
        - PACKAGE

# A list of dependencies on modules outside of the workspace. They're
# shared by all modules, and the manifest is stored in the buf.lock file.
#
# Dependencies between the modules in the workspace are automatically handled
# by Buf and shouldn't be declared here.
deps:
  - buf.build/google/googleapis
  - buf.build/grpc/grpc

# Workspace-level lint and breaking change detection settings. These settings
# are the default for all modules in the workspace unless specifically
# replaced in the module declarations.
lint:
  use:
    - DEFAULT
breaking:
  use:
    - PACKAGE

Single-module workspaces

We recommend defining module paths explicitly, but for workspaces that:

• contain only one module, and
• have a module root equal to the root of their source-control repositories

you can use a simplified version of buf.yaml. In this case, the only required field is version.

version: v2

This is logically equivalent to:

version: v2
modules:
  - path: .

You can also specify a name for this single module at the top-level to connect it to its repository on the BSR, and specify its lint, breaking, and deps values:

version: v2
name: buf.build/foo/bar
lint:
  use:
    - DEFAULT
breaking:
  use:
    - FILE
deps:
  - buf.build/googleapis/googleapis

Because it's a common use case to have a 1:1 correspondence between a module and a source control repository, we special-cased this behavior to simplify workspace setup. For all other cases, you should always specify a modules key, including when you have a single module that's contained in a subdirectory of the source control repository (such as a proto directory).

Additional requirements

The Buf CLI imposes two additional requirements on your .proto file structure for compilation to succeed, both of which are essential to successful modern Protobuf development across a number of languages.

1. Workspace modules must not overlap. A workspace module can't be a sub-directory of another workspace module.

This, for example, isn't a valid configuration:

version: v2 # THIS IS INVALID AND RESULTS IN A PRE-COMPILATION ERROR
  modules:
    - path: foo
    - path: foo/bar

Following this rule ensures that imports are consistent across all your .proto files. Without it, in the above example a file foo/bar/bar.proto could be imported as either bar/bar.proto or bar.proto. Having inconsistent imports leads to a number of major issues across the Protobuf plugin ecosystem, so we don't allow it.

2. All .proto file paths must be unique relative to each workspace module.

Consider this configuration:

version: v2
modules:
  - path: foo
  - path: bar

Given the above configuration, it's invalid to have these two files:

• foo/baz/baz.proto
• bar/baz/baz.proto

because it results in two files having the path baz/baz.proto. If you add the following file to the mix, the issue becomes apparent:

// THIS IS DEMONSTRATING SOMETHING BAD
syntax = "proto3";

package bar.baz;

import "baz/baz.proto";

Which file is being imported, foo/baz/baz.proto or bar/baz/baz.proto? With protoc the answer depends on the order of the -I flags. The Buf CLI errors out pre-compilation instead, alerting you to the issue. Though the above example is relatively contrived, vendoring .proto files is a common practice that can cause this situation.

Module and repository relationship

If the module has a corresponding repository in the BSR, that location is stored in the optional name field, which is composed of three parts: the remote, the owner, and the repository:

• Remote: The DNS name for the server hosting the BSR. This is the public BSR server unless you're on the Pro or Enterprise plan, in which case it's your private BSR server's custom domain.

• Owner: Either a user or organization within the BSR ecosystem.

• Repository: Storage for all versions of a single module.

The module's name uniquely identifies it in the BSR, which allows it to be referenced as a dependency, consumed as an SDK, and much more.

See the buf.yaml documentation for full descriptions of its configuration options, and the lint and breaking change detection documentation for full descriptions of their options and rules.

Buf operations in the workspace

Although modules are the main primitive for Buf, the workspace is the primary input for Buf operations, both locally and within CI/CD setups. This enables Buf to resolve the workspace's internal dependencies and lint/breaking change detection rules across all of its modules before building the modules and executing the requested command.

Pushing modules from workspaces

When pushing to the BSR, the workspace is the unit being pushed, as long as all of its internal dependencies have name keys that can resolve to the BSR. All modules within it are pushed to their corresponding repositories (if they exist) in dependency order. See Pushing to the BSR for more details.

Dependency management

When resolving imports, Buf looks within the workspace first—if it finds the dependency locally, it resolves it there. Otherwise, it attempts to find it in the BSR based on the declarations in the deps field. This makes it easier to iterate on related modules at the same time before pushing to the BSR.

In the workspace, imports are resolved relative to the root of the module being imported (the value of its path field), which is an assumed prefix for the import value. For example, using the same directory structure as above, assume that proto/acme/weatherapi/v1/api.proto imports both of the Protobuf files in the vendor module and calculate.proto from its own module. Its import statements would look like this:

syntax = "proto3";

package acme.weatherapi.v1;

import "units/v1/imperial.proto";
import "units/v1/metric.proto";
import "acme/weatherapi/v1/calculate.proto";

workspace_root
├─ buf.gen.yaml
├─ buf.lock
├─ buf.yaml
├─ proto
│  └─ acme
│     └─ weatherapi
│        └─ v1
│           ├── api.proto
│           ├── calculate.proto
│           └── conversion.proto
├─ vendor
│  └─ units
│     └─ v1
│        ├─ imperial.proto
│        └─ metric.proto
├─ LICENSE
└─ README.md

See Dependency management for more details.

Module cache

The Buf CLI caches files it downloads as part of module resolution in a folder on the local file system to avoid incurring the cost of downloading modules repeatedly. To choose where to cache the files, it checks these, in order:

• The value of $BUF_CACHE_DIR, if set.
• The value of $XDG_CACHE_HOME, falling back to $HOME/.cache on Linux and Mac and %LocalAppData% for Windows.

The module cache is only used for dependencies declared in the deps field. For all modules that have a value for the name field, the workspace overrides the module cache and allows you to use any new local changes. If the name either doesn't match the importing module's dependency or doesn't exist, the Buf CLI uses the module cache instead.

Referencing a module

A reference is the string used to refer to a specific version of the module, which includes the module's name and can optionally include either a commit or a label.

• If no commit or label is explicitly referenced, then the reference resolves to the latest approved commit on the repository's default label
• If the reference includes a label, it resolves to the latest approved commit associated with that label. Referencing modules by labels other than the default label is helpful when working across teams or workspaces on multiple modules so that you can depend on pre-release code while working in parallel.

The examples below show how to reference a module in the deps declaration of a buf.yaml file, but the format applies generally.

version: v2
deps:
  - buf.build/bufbuild/protovalidate

version: v2
deps:
  - buf.build/bufbuild/protovalidate:demo

version: v2
deps:
  - buf.build/bufbuild/protovalidate:f05a6f4403ce4327bae4f50f281c3ed0

Related docs

• Read about repositories and commits and labels.
• Learn more about dependency management in the BSR.
• Read the buf.yaml configuration file documentation.