Modules: Buf's key primitive
In the Buf ecosystem, a module is a collection of Protobuf files that are configured, built, and versioned as a
logical unit. By moving away from individual files, modules simplify file discovery and eliminate the need
for complex protoc build scripts to include, exclude, and configure your Protobuf sources with
Storing modules in the Buf Schema Registry (BSR), which is Protobuf-aware, also protects you from publishing broken builds. Module consumers can have confidence that the modules that they pull compile, something that isn't possible with traditional version control systems.
How modules are defined
A module is identified by a
name key in the
buf.yaml file, which is placed at
the root of the Protobuf source files it defines. This tells the Buf CLI where to search for
.proto files and how to
handle imports. Unlike protoc, where you manually specify
.proto files, the Buf CLI recursively discovers all
files defined by the configuration to build the module.
name is composed of three parts: the remote, owner, and repository:
- Remote: The DNS name for the server hosting the BSR. This is always
- Owner: An entity that is either a user or organization within the BSR ecosystem.
- Repository: Storage for all versions of a single module
The module's name uniquely identifies and gives ownership to a collection of Protobuf files, which means you can push modules to authorized repositories within the BSR, add hosted modules as dependencies, consume modules as part of code generation, and much more.
Organizing and managing modules
To create a Buf module, you should start by collecting one or more related Protobuf packages within a module, each serving a specific and valuable function. For instance, you can create a module consisting of packages with APIs for financial analysis to enable other developers building financial applications to use your work.
The Protobuf language organizes packages, and packages group into buf modules. Your module declares dependencies required to run your code, including the set of other modules it needs.
How modules map to the buf.yaml file
As you add new features or improve the functionality of your module, you can release updated versions. This allows developers using your module to import the most recent packages and test them before deploying them to production. They can make RPC calls to your module and experiment with the new version, making sure it meets their requirements before using it in production.
Module best practices
Though the Buf Schema Registry (BSR) automatically enforces module compilation upon pushing, there are additional best practices that are worth considering during module development. In this context, we'll discuss these best practices and explain their importance.
Organizing code in the repository
By adhering to the conventions outlined in this guide, you can streamline maintenance and enhance the development experience for your module. Incorporating your module code into a repository is usually as straightforward as it is with other code. To further clarify, please refer to the diagram below, which demonstrates a source hierarchy for a simple module containing two packages.
│ └── pkg
│ └── v1
│ └── pkg.proto
|The module's license.
|The buf.md file is similar to the README.md file found in GitHub repositories, and it currently accommodates all the syntax provided by CommonMark.
|Describes the module, including its module path (which effectively serves as its name) and its dependencies. For additional information, please refer to the buf.yaml reference. The module name will be specified using a name directive, as shown in the following example: name: buf.build/connectrpc/eliza. For further guidance on selecting an appropriate module path, please consult the Managing Dependencies section.
|This file includes the module's dependencies, which buf uses to manage the dependent modules and their versions. The file will be empty or non-existent if there are no dependencies. It's not recommended to manually modify this file, except when using the buf mod update command.
|Package directories and
.proto files that make up the Protobuf packages and sources within the module.
While the module is already a versioned entity composed of Protobuf files, it's advisable to implement some level of versioning in its directory and package structure as well.
For example, suppose you're creating the buf.build/acme/pkg module, which initially contains a single
.proto file. In
that case, it's preferable to nest this file within a directory and define a unique package that differentiates it from
other module dependencies, rather than placing it at the root of the module (alongside the buf.yaml and buf.lock files).
Failing to adhere to this best practice increases the likelihood of API collisions with other user-defined APIs. For
instance, if a consumer intends to import Protobuf definitions from two modules that both define an
api.proto file, the
resulting module won't compile. This is because the compiler can't differentiate between which
api.proto file to
reference if there are multiple versions of it.
The module layout described here is included in the
In addition to comments associated with your Protobuf definitions, it's necessary to have a way for module authors to
describe the module's functionality for others to understand. To achieve this, you can create a
buf.md file in the same
directory as the module's
buf.yaml file and push it to the BSR in the usual manner. As documentation is an integral part
of the module, any changes made to the
buf.md file result in new commits in the BSR.
buf.md file is similar to a GitHub repository's
README.md and currently supports all CommonMark syntax.
Alternatively, a README.md (or README.markdown) file in a module is used as the module documentation, if buf.md is not available.
If both buf.md and README.md files are available, buf.md is considered the module documentation.
│ └── pkg
│ └── v1
│ └── pkg.proto
Public repositories on the Buf Schema Registry are often used to share open source software. For your repository to truly be open source, you'll need to license it so that others are free to use, change, and distribute the software.
As a best practice, we encourage you to include the license file with your project. To do this, simply include
LICENSE in the same directory as your module's
buf.yaml file and push it to the BSR.
Buf aims to provide users with open source license information to help them make informed decisions. However, we are not legal experts and do not guarantee the accuracy of the information provided. We recommend consulting with a professional for any legal issues related to open source licenses.
Referencing a module
A module has different versions. Each version includes any number of changes, and each change is identifiable by a unique commit, tag, or branch. The collective set of module versions is housed within a repository.
Commit: Every push of new content to a repository is associated with a commit that identifies that change in the schema. The commit is created after a successful push. This means that unlike Git, the commit only exists on the BSR repository and not locally.
Tag: A reference to a single commit but with a human-readable name, similar to a Git tag. It is useful for identifying commonly referenced commits—like a release.
Branch: A series of commits in a development workflow with a human-readable name, similar to a Git branch. Branches are useful for iterating on a module while keeping those changes outside of the main branch. When a branch name is used as a dependency in
buf.yamlof a module, the module cannot be pushed until you update it to reference on the main branch of the dependency.
Local modules with workspaces
If you want to depend on local modules, you can set up a
workspace to discover modules through your file
system. If you are in a workspace,
buf looks for
deps in your
workspace configuration before
attempting to find it on the BSR.
This makes workspaces a good way to iterate on multiple modules at the same time before pushing any changes to the BSR.
The Buf CLI caches files it downloads as part of module resolution in a folder on the local filesystem to avoid incurring the cost of downloading modules repeatedly. To choose where to cache the files, it checks these, in order:
- The value of
$BUF_CACHE_DIR, if set.
- The value of
$XDG_CACHE_HOMEfalling back to
$HOME/.cacheon Linux and Mac and