Overview

One of the core promises of Protobuf is forwards and backwards compatibility. However, making sure that your Protobuf schema doesn't introduce breaking changes isn't automatic - there are rules you need to follow to ensure your schema remains compatible for the lifetime of it's evolution.

Buf provides a breaking change detector through buf check breaking, which runs a set of breaking checkers across the current version of your entire Protobuf schema in comparison to a past version of your Protobuf schema. The checkers are selectable, and split up into logical categories depending on the nature of breaking changes you care about:

  • FILE: Generated source code breaking changes on a per-file basis, that is changes that would break the generated stubs where definitions cannot be moved across files. This makes sure that for languages such as C++ and Python where header files are included, your source code will never break for a given Protobuf change. This category also verifies wire and JSON compatibility.
  • PACKAGE: Generated source code breaking changes on a per-package basis, that is changes that would break the generated stubs, but only accounting for package-level changes. This is useful for languages such as Java (with option java_multiple_files = true; set) or Golang where it is fine to move Protobuf types across files, as long as they stay within the same Protobuf package. This category also verifies wire and JSON compatibility.
  • WIRE: Wire breaking changes, that is changes that would break wire compatibility, including checks to make sure you reserve deleted types of which re-use in the future could cause wire incompatibilities.
  • WIRE_JSON: Wire breaking changes and JSON breaking changes, that is changes that would break either wire compatibility or JSON compatibility. This mostly extends WIRE to include field and enum value names.

Buf can compare your current Protobuf schema against different Input types:

  • Stored Images or FileDescriptorSets from protoc, representing your past schema. These Images can be in either local or remote (http/https) locations.
  • Branches or tags of your git repository, either local or remote. Buf can clone the head of the branch, and compile the .proto files in-memory on that branch to compare against, on the fly.
  • Tar or zip archives, either local or remote, containing your past .proto files. Buf then compiles these files in-memory on the fly and compares against them.

Other features of buf check breaking include:

  • Automatic file discovery. By default, Buf will build your .proto files by walking your file tree and building them per your build configuration. This means you no longer need to manually specify your --proto_paths and files every time you run the tool. However, Buf does allow manual file specification through command-line flags if you want no file discovery to occur, for example in Bazel setups.

  • File references. Buf's breaking change detector will produce file references to the location of the breaking change, including if a reference moves across files between your past and current file versions. For example, if a field changes type, Buf will produce a reference to the field. If a field is deleted, Buf will produce a reference to the location of the message in the current file.

  • Selectable error output. By default, Buf outputs file:line:col:message information for every breaking change, with the file path carefully outputted to match the input location, including if absolute paths are used. JSON output that includes the end line and end column of the breaking change is also available, and JUnit output is coming soon.

  • Speed. Buf's internal Protobuf compiler utilizes all available cores to compile your Protobuf schema, while still maintaining deterministic output. As an unscientific example, Buf can compile all 2,311 .proto files in googleapis in about 0.8s on a four-core machine, as opposed to about 4.3s for protocon the same machine. While both are very fast, this allows for instantaneous feedback. A typical breaking check operation for 2,300 .proto files that clones the head of a local git repository branch, compiles all the .proto files on that branch, compiles all the current local files, and compares the two results for breaking changes, takes about 2.2s.

  • Use protoc as your compiler. Existing lint and breaking change detection tools produce an internal representation of your Protobuf schema in one of two ways:

    • By using a third-party Protobuf parser, which is usually error-prone and almost never covers every edge case of the Protobuf grammar.
    • By shelling out to protoc itself and parsing the result, which not only requires specific management of protoc in relation to the lint/breaking change detection tool, but can be cumbersome and error-prone itself, especially if the tool parses error output from protoc.

    Buf tackles this issue by using FileDescriptorSets (which we extend into Images) internally for all operations, and allowing these FileDescriptorSets to be produced in one of two ways:

    • By using a newly-developed Golang Protobuf compiler that is continuously tested against thousands of known Protobuf definitions, including all known edge cases of the Protobuf grammar.
    • By allowing users to provide protoc output as buf input, thereby bypassing any compiling or parsing on the part of buf entirely, and instead using protoc, the gold standard of Protobuf compilation.

    See the Image and compiler documentation for more details.

    In short, we don't expect you to natively trust the internal compiler is actually equivalent to protoc - we would want to verify this claim ourselves. There are also cases (such as Bazel setups) where you may already have infrastructure around calling protoc, and may want to just use artifacts from protoc as input to buf.

  • Use buf as a protoc plugin instead of a standalone tool. You can go a step further and use Buf's breaking change detection functionality as a protoc plugin with the provided protoc-gen-buf-check-breaking plugin.