Manage dependencies

In this article, we will focus on how buf mod manages dependencies for a Protobuf project. This is a crucial aspect of any project as it ensures that all required dependencies are present and up-to-date, ensuring compatibility and avoiding conflicts.

Dependencies in a Protobuf project

A Protobuf project can have dependencies on other Protobuf files, libraries, or modules. These dependencies are generally imported using the import statement in a Protobuf file. Managing these dependencies is important to ensure the correct versions are being used, to prevent conflicts, and to simplify the maintenance of the project.

buf.yaml and buf.lock Files

buf uses two files to manage dependencies: buf.yaml and buf.lock.

1. buf.yaml: This file is used to specify the project's direct dependencies, as well as other configuration settings, such as lint and breaking change rules. The dependencies are specified in the deps section, where each dependency is represented by a module reference on the BSR.

version: v1
deps:
  - buf.build/bufbuild/eliza
lint:
  use:
    - DEFAULT
breaking:
  use:
    - FILE

2. buf.lock: This file is automatically generated and managed by buf and contains the exact versions of all direct and transitive dependencies used in the project. This ensures reproducible builds and makes it easy to keep track of dependency versions.

# Generated by buf. DO NOT EDIT.
version: v1
deps:
  - remote: buf.build
    owner: bufbuild
    repository: eliza
    commit: f3801d450ef94549afec851bc73de581

Although we do not recommend it, in some situations you may need to pin a module to a specific version. You can do this by specifying a tag, a commit name, or a draft name in your deps after the : delimiter. Ideally, authors keep modules backwards-compatible and avoid breaking changes, so you can always rely on the latest version.

deps:
  - buf.build/bufbuild/eliza:1c473ad9220a49bca9320f4cc690eba5

Managing Dependencies with buf mod

buf mod provides several commands to manage dependencies in a Protobuf project:

• buf mod init: Initializes the buf.yaml file in the current directory. It is used to set up a new project or to convert an existing project to use buf mod.
• buf mod update: Updates the buf.lock file to reflect the latest versions of the direct dependencies specified in the buf.yaml file. If a new dependency is added to the buf.yaml file, running buf mod update will resolve and lock the new dependency's version in the buf.lock file.
• buf mod prune: Removes any unused dependencies from the buf.lock file. This is useful to keep the dependency list clean and up-to-date.

Resolving Protobuf Imports

When a Protobuf file imports another Protobuf file, the import statement refers to the target file by its path. This import resolution process involves locating the target file in the project or within its dependencies. Understanding how Protobuf imports are resolved is essential for properly managing dependencies and avoiding conflicts.

Import Resolution Process

The import resolution process can be broken down into several steps:

Local imports

The resolution process first checks for imports relative to a buf.yaml in its parents directories. If the specified import exists relative to the buf.yaml, the imported file is resolved locally.

For example, consider a project with the following structure:

buf.yaml
├── bar
│   └── bar.proto
└── foo
    └── foo.proto
    └── foo2.proto

A valid import statement within the foo/foo.proto is resolved relative to the buf.yaml file:

import "bar/bar.proto";

If local files want to import neighboring files they still need to import relative to the buf.yaml file. This means that all import statements would be the same to import any file in a project.

// This import is invalid
import "foo.proto";
// This is correct
import "foo/foo.proto";

Module imports

If the resolution process fails to find a file locally the process will check the deps listed in the buf.yaml file. It searches through the dependencies in the order they are listed. If the import is found within one of the modules then the import is resolved.

For example, the same project might depend on a module buf.build/bufbuild/eliza as specified in the buf.yaml file:

version: v1
deps:
  - buf.build/bufbuild/eliza

If there is an import statement like:

import "connect/demo/eliza/v1/eliza.proto";

The resolution process will first search for the connect/demo/eliza/v1/eliza.proto locally, fail, then continue to searching within the buf.build/bufbuild/eliza module for the same import path.

If the import cannot be resolved using local imports or module imports, the resolution process fails, and an error is reported.

Tips for Managing Imports

• Use clear and consistent import paths: Avoid using generic names for your Protobuf files and organize them in a clear directory structure. This helps prevent conflicts and makes imports easier to understand.

• Avoid circular imports: Circular imports can cause problems in the resolution process and should be avoided. Instead, refactor your Protobuf files to remove circular dependencies.

• Keep your dependencies up to date: Use the buf mod subcommands to manage your dependencies and ensure they are always up to date, preventing conflicts and ensuring compatibility.

• Adhere to The Official Buf Style Guide and use buf breaking, buf lint, and buf format to keep your Protobuf files in top-notch shape.

By understanding the import resolution process and following best practices, developers can effectively manage Protobuf imports and dependencies, ensuring a well-structured and maintainable project.

Tamper-proofing dependencies in the BSR

The Buf CLI and the BSR are equipped to keep your workflows secure from tampering attacks, allowing you to trust your dependencies on every module build.

What does a tampering attack look like?

Tampering means someone has made unauthorized changes in your dependencies — more exactly, in Protobuf schemas you depend upon. Imagine that your application depends on a BSR module that defines messages for moving money between two accounts. A nefarious actor could swap the tags of which account is being debited and which is being credited. Making a payment becomes getting paid. And it’s visually difficult to detect:

message MoveMoney {
-  string from_email = 1;
+  string from_email = 2;
-  string to_email = 2;
+  string to_email = 1;
  google.type.Money amount = 3;
}

Our tamper resistance technique is a typical solution to this problem: use a cryptographic hash and keep a record of the content’s digest. If the content changes, so will the digest. And if the CLI detects that a downloaded dependency digest is different than the expected one, it warns loudly that something is wrong, and avoids using that dependency.

How the BSR protects your dependencies

Every time a module is pushed to a BSR repository, the CLI sends the source files to the BSR as separated blobs with their calculated hash, along with a module’s manifest that includes a canonical list of all of the files and the digest that are being pushed. The BSR receives these separated blobs and manifest and stores them as CAS (Content-Addressable-Storage), making sure that the generated commit points to the manifest, and this manifest links to all of the files that were pushed in the original request.

After that, when a pushed module is used as a dependency in another repository, the original manifest digest is recorded in the buf.lock file, so you can check the cryptographic digest of each dependency into source control. For example:

# Generated by buf. DO NOT EDIT.
version: v1
deps:
  - remote: buf.build
    owner: bufbuild
    repository: eliza
    commit: f3801d450ef94549afec851bc73de581
    # Buf CLI watches out for changes in this field
    digest: shake256:cd74f1dc4de0c750b7cf43df018b368a565de21a748738967c9a630aec95b5f267731a74a1ba142d7dbd2745300976671f9d90aa0a472b1f5b763fd886c66c12

How do we know if/when a dependency has been tampered with?

Whenever you’re building your modules, buf build examines your local Buf cache, downloads any missing dependencies, and checks its manifest digest against the one stored in the buf.lock file. The CLI warns you if they don’t match and fails the build, avoiding use of the offending dependency consumption.

Such a scenario would mean that the dependency had different content than the one defined in the lock file, which could mean a potential MITM (Man in the middle) attack.

As long as your buf.lock is checked into source control, and you keep your CLI version regularly updated, dependencies consumed from the BSR are protected against tampering.

Conclusion

buf mod plays a crucial role in managing dependencies for a Protobuf project. It simplifies the process of maintaining and updating dependencies by automating the resolution and locking of dependency versions. By using buf mod along with the other features provided by the buf command-line tool, developers can ensure their Protobuf projects are maintainable, compatible, and follow best practices.

For examples of how to manage dependencies see how to create and push a module and Update Protobuf modules with buf and the BSR