Prototool

Prototool is a widely-used Protobuf tool that has a builder, linter, formatter, breaking change detector, gRPC CLI, and configurable plugin executor.

In this document, we'll discuss the pros and cons of Prototool vs Buf's build, lint and breaking change detection functionality, as well as Buf equivalent commands and migration.

Prototool Pros

  • Prototool has a formatter via prototool format. Buf does not have a formatter as of now - See our discussion for more details. The short is that Protobuf linters as they are written now either rely on third-party Protobuf parsers (as Prototool does), which can result in corrupt files, or on FileDescriptorSets, which are lossy. However, if you're willing to assume the risks, this is functionality that Prototool has that Buf does not.
  • Prototool has a configurable plugin executor via prototool generate. Buf does not have this feature - we think that Protobuf stub generation as it is now has numerous problems, and are concentrated on the Buf Schema Registry. However, for users that locally generate stubs with Prototool, we provide a mitigation strategy with a simple bash command below that effectively replicates most of prototool generate.
  • Prototool has gRPC CLI functionality via prototool grpc. This functionality roughly models parts of grpcurl but with fewer available features. We think gRPC CLI functionality is better left to gRPC-specific tooling for now. See our gRPC documentation for how to use Buf Images as grpcurl input.
  • Prototool has a much more prescriptive set of lint rules via the uber2 lint group. This is a much more opinionated set of lint rules than Buf's DEFAULT category. Having a more opinionated set of lint rules is helpful for maintaining a very high level of consistency across a company's Protobuf schema, and we are working on a STRICT category for Buf which will have additional lint checkers. We feel that the DEFAULT category is a set of checkers that universally applies to many existing Protobuf schemas.
  • Prototool provides .proto template generation, specific to the uber1 and uber2 lint groups, via prototool create. There is no equivalent functionality in Buf and we do not have plans to provide such functionality.

Prototool Cons

  • By far the biggest con of Prototool is that it both uses uses a third-party Protobuf parser that is not tested to cover every edge case of the Protobuf grammar, while additionally shelling out to protoc to verify that files are valid. The third-party Protobuf parser Prototool uses has had issues in the past with breakages, and as this parser does not verify that what it is parsing is actually valid Protobuf, Prototool shells out to protoc to verify validity. This means that Prototool is susceptible to both breakages for valid Protobuf files (if the parser fails), as well has having all the drawbacks of shelling out to protoc, especially parsing of protoc output. Prototool attempts to parse stderr from protoc output, which has breaking changes across minor versions of protoc. By default, Prototool downloads protoc for you, which is helpful for many cases, but can cause issues if the download fails, the cache is corrupted, or if the protoc version is not locked. We highly recommend reading our discussion on Protobuf compilation for more details.

    Instead, Buf lets you use either the internal compiler that is tested to cover every edge case and will only parse valid files, or use protoc output as buf input. Buf can actually use many types of input, including protoc output, local or remote git repositories, and local or remote archives. Buf never shells out to external commands to perform any of it's functionality. Buf also has no cache as it does not need to cache any external binaries to perform it's functionality.

  • Prototool runs file discovery for your Protobuf files, but provides no mechanism to skip file discovery and specify your files manually, outside of running commands for files one at a time, which breaks some lint and breaking change detection functionality. Buf allows you to skip file discovery and specify your files manually for use cases that require this, such as Bazel.

  • Prototool performs configuration file discovery - Prototool will recursively go up directories until it finds a configuration file. This may seem like a feature, but leads to issues where a configuration file may be in i.e. your home directory, which affects how building and linting works for an unsuspecting checked out repository in some sub-directory of your home directory. Even further, Prototool performs Protobuf file discovery for almost every command, meaning that running commands like prototool lint --list-all-linters can fail or time out because of running this command in a location with many files, for example your home directory. Instead, Buf only looks at your current directory for a configuration file, while allowing you to specify alternate configuration file locations, or configuration data on the command line (Prototool also allows configuration data on the command line). This means your builds are always deterministic no matter what your local computer's file structure is.

  • Prototool's lint functionality lets you select a single group, currently google, uber1, or uber2, and then add and remove rules from that specific group. Buf instead provides lint categories that you can mix and match, and lets you exclude entire categories or checkers if you want. Buf also presents a clear path to add additional checkers to new categories in a backwards-compatible manner without touching existing categories.

  • Prototool's breaking change detector cannot be configured as to what rules it runs to verify breaking change detection. Buf's checkers are fully configurable, including ignores on a per-directory or per-file basis for every breaking checker or category.

  • Breaking change rules are not a binary proposition - there are different kinds of breaking changes that you may care about. Buf provides four categories of breaking change rules to select - per-file generated stub breaking changes, per-package generated stub breaking changes, wire breaking changes, and wire + JSON breaking changes. Within these categories, you can go further and enable or disable individual checkers through configuration. Prototool effectively only checks per-package generated stub breaking changes.

  • Prototool does not cover all possible issues per the FileDescriptorSet definition of what is a breaking change, even for per-package generated stub breaking changes.

  • Buf provides file:line:column:message references for breaking change violations, letting you know where a violation occurred, including potentially integrating this into your editor in the future. These reference your current Protobuf schema, including if types move across files between versions of your Protobuf schema. The error output can be outputted as text or JSON, with other formats coming in the future. Prototool prints out unreferenced messages.

  • Since Buf can process FileDescriptorSets as input, Buf provides protoc plugins protoc-gen-buf-check-lint and protoc-gen-buf-check-breaking to allow you to use Buf's lint breaking change detection functionality with your current protoc setup.

Prototool lint groups to Buf lint categories

Buf has lint categories that are either roughly equivalent or a subset of Prototool lint groups. Buf does not have linting functionality some elements such as file option naming, although we will be adding a STRICT category in the future. See the "what we left out" documentation for more details.

google

The following Prototool configuration:

lint:
group: google

Is equivalent to the following Buf configuration:

lint:
use:
- STYLE_BASIC
except:
- ONEOF_LOWER_SNAKE_CASE
- PACKAGE_LOWER_SNAKE_CASE

However, we recommend using one of the "top-level categories" MINIMAL, BASIC, or DEFAULT instead. See the lint checkers documentation for more details.

uber1, uber2

The uber1 and uber2 Prototool lint groups are supersets of the DEFAULT Buf lint category, except you will need to set overrides for enum value and service suffixes. That is, buf check lint should pass for all Protobuf schemas (except as discussed below) that use uber1 or uber2 with Prototool, given the following Buf configuration:

lint:
use:
- DEFAULT
enum_zero_value_suffix: _INVALID
service_suffix: API

The only exception to this is for nested enum values with the uber1 lint group. The uber1 lint group expects the enclosing message name for enums to be part of enum value names. For example, this is a valid nested enum for uber1:

// THIS IS FOR UBER1 IN PROTOTOOL
// THIS WILL NOT PASS BUF'S ENUM_VALUE_PREFIX LINT CHECKER
message Foo {
enum Bar {
FOO_BAR_INVALID = 0;
FOO_BAR_ONE = 1;
}
}

For the uber2 lint group, and for Buf, the enclosing message name should not be part of the enum value prefix. While Prototool's lint rule allows uber1-style prefixes for backwards compatibility, Buf expects that the prefix only include the enum name. For example:

message Foo {
enum Bar {
BAR_INVALID = 0;
BAR_ONE = 1;
}
}

Protobuf allows multiple enum values with the same name in different nested messages - this does not violate the scoping rules.

Configuration

See the following documentation for more details on configuration:

Buf primarily uses a buf.yaml configuration file that should be at the root of your repository for configuration. This roughly corresponds to the prototool.yaml configuration file for Prototool. We'll discuss the Prototool configuration sections below.

excludes

Corresponds to build.excludes in Buf.

Note that Buf also allows you to configure sub-directories to search with build.roots, as opposed to Prototool where this is done implicitly by the location of your prototool.yaml file with no override option available.

protoc

There is no equivalent in Buf.

Buf does not download or shell out to protoc. However, protoc.includes corresponds to adding such a value to build.roots in Buf. See the build configuration documentation for more details.

create

There is no equivalent in Buf.

Buf does not have .proto template generation.

lint.group

Corresponds to lint.use in Buf.

Buf allows you to specify categories or ids in lint.use, while lint.group in Prototool only specifies the single group to use as a base set of rules.

lint.ignores

Corresponds lint.ignore_only in Buf.

Buf also allows you to ignore all checkers for specific directories through lint.ignore. Note that paths for Buf's lint.ignore and lint.ignore_only are relative to your roots, see the lint configuration documentation for more details.

lint.rules

Corresponds to lint.use and lint.except in Buf.

See the lint configuration documentation for more details.

lint.file_header

There is no equivalent in Buf.

Buf does not have a formatter, and this option is largely integrated into the formatter. We can provide file header functionality in the future if there is a big demand for it, but this is probably best left to other tooling.

lint.java_package_prefix

There is no equivalent in Buf.

Buf does not check file options as of now, see our discussion on this for more details.

break.include_beta

Corresponds to break.ignore in Buf, effectively.

Buf will check all packages by default for breaking changes. By default, Prototool ignores packages that match v\d+beta\d+$. Buf instead lets you specify any packages you want to ignore through break.ignore, and also allows you to ignore specific categories or checkers through break.ignore_only. See the breaking configuration documentation for more details.

break.allow_beta_deps.

There is no equivalent in Buf.

Buf does not do package dependency enforcement, although we could add this feature in a more generic fashion through a new buf check sub-command in the future if there is a demand for it.

generate

There is no equivalent in Buf.

Buf does not handle local stub generation. See below for mitigation if this is a feature you rely on in Prototool.

Equivalent commands

prototool all

There is no equivalent in Buf.

The command prototool all runs formatting and linting at once. It doesn't present an easy manner to extend what the definition of "all" means, for example breaking change detection. Since Buf is relatively fast in it's various functionality (for example, compiling and linting all 2,311 files in googleapis takes about 0.8s with Buf), we feel that it is better to run multiple commands for the functionality you want to perform.

prototool break check --git-branch master

buf check breaking --against-input '.git#branch=master'
buf check breaking --against-input '.git#tag=v1.0.0'

Prototool's --json flag can be replaced with --error-format=json with Buf.

prototool break check --descriptor-set-path lock.bin

buf check breaking --against-input lock.bin

prototool cache

There is no equivalent in Buf.

Buf does not have a cache, as it does not shell out to external commands.

prototool compile

$ buf image build -o /dev/null

Buf handles /dev/null on Mac and Linux (and nul for the future in Windows) as a special-case, and even though writing to /dev/null is very quick, Buf stops short on writing if this is specified.

prototool config init

We think there is no need to have a special command for this. :-)

prototool create

There is no equivalent in Buf.

Buf does not do .proto template generation.

prototool descriptor-set

$ buf image build --exclude-imports --exclude-source-info -o -

This writes a binary Image to stdout. While Images are wire-compatible with FileDescriptorSets f you want to strip the extra metadata, you can do so with the --as-file-descriptor-set flag. If you want to write to a file, specify the file path for -o instead of -.

prototool files

$ buf ls-files

prototool format

There is no equivalent in Buf.

Buf does not have a formatter, see the above discussion for more details.

prototool generate

There is no equivalent in Buf, however you can easily replace most of this functionality with bash. Prototool does do additional work for protoc-gen-go and protoc-gen-gogo, by setting the Mkey=value options automatically, but this is generally not needed. We'll use java as an example, however. Given the following Prototool configuration:

generate:
plugins:
- name: java
output: gen/java

The following replicates prototool generate by running protoc on a per-directory basis, which some plugins (for example protoc-gen-go) effectively require:

mkdir -p gen/java
for dir in $(find . -name '*.proto' -print0 | xargs -0 -n1 dirname | sort | uniq); do
protoc -I . --java_out=gen/java $(find "${dir}" -name '*.proto')
done

This is also done via our protoc_gen_plugin.bash script we use until the Buf Schema Registry is ready, however you should copy that script instead of pulling directly from GitHub as we make no guarantee it will exist in the long-term.

Per the compiler documentation, you could go a step further by using Buf's build configuration and internal compiler (however this should be considered experimental functionality):

$ mkdir -p gen/java
$ buf image build -o - | protoc --descriptor_set_in=/dev/stdin --java_out=gen/java $(buf image build -o - | buf ls-files --input -)

We recommend managing your own stub generation instead of using prototool generate. There are a lot of issues with stub generation as it is now, and we're aiming to fix that with the Buf Schema Registry. For now, we recommend invoking protoc yourself, or using Bazel.

prototool grpc

There is no equivalent in Buf.

Buf does not have gRPC functionality, as discussed above. We recommend using grpcurl instead. See the gRPC documentation for details on how to use FileDescriptorSets produced by Buf for grpcurl input.

prototool lint

$ buf check lint

Prototool's --json flag can be replaced with --error-format=json with Buf.

prototool lint --list-linters

$ buf check ls-lint-checkers

prototool lint --list-all-linters

$ buf check ls-lint-checkers --all

prototool version

$ buf --version

prototool x inspect

There is no equivalent in Buf.

We recommend using buf image build -o -#format=json | jq instead for Protobuf schema inspection. We will likely provide additional tooling for inspection in the future through a different mechanism.

Docker

Prototool provides a Docker image with prototool installed. The equivalent Docker image for Buf is bufbuild/buf. For example:

docker pull bufbuild/buf
docker run --volume "$(pwd):/workspace" --workdir "/workspace" bufbuild/buf check lint

Note that the buf command is the ENTRYPOINT, so you omit buf from the docker run invocation.