Dependency management
Buf modules can depend on other modules, which you specify in the module's buf.yaml
file.
Once you've specified a dependency, you pin your module to it in the buf.lock
file, and you can then import any
.proto
file that the dependency contains.
Using buf mod
to manage dependencies
The Buf CLI's buf mod
subcommands provide several commands to manage dependencies in your modules:
buf mod init
: Initializes abuf.yaml
file in the current directory. It's used to set up a new project or to convert a directory of.proto
files into a Buf module.buf mod update
: Updates thebuf.lock
file to reflect the latest versions of the dependencies specified in thebuf.yaml
file. If a new dependency is added to thebuf.yaml
file, running this command will resolve and lock the new dependency's version in thebuf.lock
file.buf mod prune
: Removes any unused dependencies from thebuf.lock
file. This is useful to keep the dependency list clean and up to date.
buf.yaml
and buf.lock
files
Buf uses two files to manage dependencies: buf.yaml
and buf.lock
.
The buf.yaml
file is used to specify the project's direct dependencies and other configuration settings. You specify
your dependencies in the deps
section, where each dependency is represented by a reference to a module on the BSR.
version: v1
deps:
- buf.build/connectrpc/eliza
lint:
use:
- DEFAULT
breaking:
use:
- FILE
Although we don't recommend it, in some situations you may need to pin a module to a specific version. You can do
this by specifying a tag, a commit ID, or a branch name in your deps
after the :
delimiter. Ideally, authors keep
modules backwards-compatible and avoid breaking changes, so you can always rely on the latest version.
deps:
- buf.build/connectrpc/eliza:d8fbf2620c604277a0ece1ff3a26f2ff
The buf.lock
file is automatically generated and managed by Buf, and contains the exact versions of all direct and
transitive dependencies used in the module.
# Generated by buf. DO NOT EDIT.
version: v1
deps:
- remote: buf.build
owner: connectrpc
repository: eliza
commit: d8fbf2620c604277a0ece1ff3a26f2ff
How Buf resolves imports
Understanding how imports are resolved is essential for properly managing dependencies and avoiding conflicts. When a
Protobuf file imports another Protobuf file, the import
statement refers to the target file by its path. Buf's import
resolution process involves locating the target file in the module or within its dependencies, and has two steps:
resolving local imports and resolving module imports.
Local imports
The resolution process first checks the module's .proto
files for imports relative to its buf.yaml
file. If the
specified import exists relative to the buf.yaml
file, the imported file is resolved locally.
For example, consider a module with the following structure:
buf.yaml
├── bar
│ └── bar.proto
└── foo
└── foo.proto
└── foo2.proto
A valid import statement within the foo/foo.proto
is resolved relative to the buf.yaml
file:
import "bar/bar.proto";
Local files that import neighboring files still need to import relative to the buf.yaml
file. This means that all
import
statements for a file in the module are identical regardless of the importing .proto
file's location. For
example, given the module structure above:
// This is invalid
import "foo.proto";
// This is correct
import "foo/foo.proto";
Module imports
If the resolution process fails to find a file locally, it will check the deps
listed in the buf.yaml
file. It
searches through the dependencies in the order they're listed. For example, a .proto
file might depend on the
buf.build/googleapis/googleapis
module as specified in the buf.yaml
file:
version: v1
deps:
- buf.build/googleapis/googleapis
If the .proto
file contains an import statement like:
import "google/type/datetime.proto";
The resolution process first searches for the google/type/datetime.proto
locally, and if it fails, then searches
within the buf.build/googleapis/googleapis
module for the same import path. If the import can't be resolved using
local imports or module imports, the resolution process fails and Buf returns an error.
Tips for managing imports
-
Use clear and consistent import paths. Avoid using generic names for your Protobuf files and organize them in a clear directory structure. This helps prevent conflicts and makes imports easier to understand.
-
Avoid circular imports. Instead, refactor your Protobuf files to remove circular dependencies.
-
Keep your dependencies up to date. Use the
buf mod
subcommands to manage your dependencies and ensure they're always up to date, preventing conflicts and ensuring compatibility. -
Adhere to the Buf style guide and use
buf breaking
,buf lint
, andbuf format
to keep your Protobuf files in top-notch shape.
By understanding the import resolution process and following best practices, you can effectively manage Protobuf imports and dependencies, ensuring a well-structured and maintainable project.
Tamper-proofing dependencies in the BSR
The Buf CLI and the BSR are equipped to keep your workflows secure from tampering attacks, allowing you to trust your dependencies on every module build.
What does a tampering attack look like?
Tampering means someone has made unauthorized changes in your dependencies—more exactly, in Protobuf schemas you depend upon. Imagine that your application depends on a BSR module that defines messages for moving money between two accounts. A nefarious actor could swap the tags of which account is being debited and which is being credited. Making a payment becomes getting paid. And it’s visually difficult to detect:
message MoveMoney {
- string from_email = 1;
+ string from_email = 2;
- string to_email = 2;
+ string to_email = 1;
google.type.Money amount = 3;
}
Our tamper resistance technique is a typical solution to this problem: use a cryptographic hash and keep a record of the content’s digest. If the content changes, so will the digest. And if the CLI detects that a downloaded dependency digest is different than the expected one, it warns loudly that something is wrong, and avoids using that dependency.
How the BSR protects your dependencies
Every time a module is pushed to a BSR repository, the CLI sends the source files to the BSR as separated blobs with their calculated hash, along with a module’s manifest that includes a canonical list of all of the files and the digest that are being pushed. The BSR receives these separated blobs and manifest and stores them as CAS (Content-Addressable-Storage), making sure that the generated commit points to the manifest, and this manifest links to all of the files that were pushed in the original request.
Files that are pushed on a buf push
run:
- Configuration file:
buf.yaml
, orbuf.mod
(deprecated) - Dependencies lock file:
buf.lock
- Module’s license:
LICENSE
- Module’s documentation:
buf.md
, orREADME.md
- All of your
*.proto
files.
For a deep dive on these files, see Buf’s source management docs.
After that, when a pushed module is used as a dependency in another repository, the original manifest digest is recorded
in the buf.lock
file, so you can check the cryptographic digest of each dependency into source control. For example:
# Generated by buf. DO NOT EDIT.
version: v1
deps:
- remote: buf.build
owner: bufbuild
repository: eliza
commit: f3801d450ef94549afec851bc73de581
# Buf CLI watches out for changes in this field
digest: shake256:cd74f1dc4de0c750b7cf43df018b368a565de21a748738967c9a630aec95b5f267731a74a1ba142d7dbd2745300976671f9d90aa0a472b1f5b763fd886c66c12
How do we know if/when a dependency has been tampered with?
Whenever you’re building your modules, buf build
examines your local Buf cache, downloads any missing dependencies,
and checks its manifest digest against the one stored in the buf.lock
file. The CLI warns you if they don’t match and
fails the build, avoiding use of the offending dependency consumption.
Such a scenario would mean that the dependency had different content than the one defined in the lock file, which could mean a potential MITM (Man in the middle) attack.
As long as your buf.lock
is checked into source control, and you keep your CLI version regularly updated, dependencies
consumed from the BSR are protected against tampering.
Related docs
For examples of how to manage dependencies, see how to create and push a module and Update Protobuf modules with buf and the BSR