Buf's goal is to shift API development toward a schema-driven paradigm and thus pave the way for a future in which APIs are defined in a way that service owners and clients can depend on.

Defining APIs using an IDL provides a number of benefits over simply exposing REST/JSON services, and today, Protobuf is the most stable, widely adopted IDL in the industry. But as things stand today, using Protobuf is much more difficult than using JSON as your data transfer format.

Buf is building tooling to make Protobuf reliable and user friendly for service owners and clients, while keeping it the obvious choice on the technical merits. Your organization should not have to reinvent the wheel to create, maintain, and consume Protobuf APIs efficiently and effectively. We'll handle your Protobuf management strategy for you, so you can focus on what matters.

The problems we aim to solve

Traditionally, adopting Protobuf presents a number of challenges across the API lifecycle. These are the problems we aim to solve:

• API designs are often inconsistent: Writing maintainable, consistent Protobuf APIs isn't as widely understood as writing maintainable REST/JSON-based APIs. With no standards enforcement, inconsistency can arise across an organization's Protobuf APIs, and design decisions can inadvertently affect your API's future iterability.

• Dependency management is usually an afterthought: Protobuf files are vendored manually, with an error-prone copy-and-paste process from GitHub repositories. Before the BSR, there was no centralized attempt to track and manage around cross-file dependencies. This is analogous to writing JavaScript without npm, Rust without cargo, Go without modules, and all of the other programming language dependency managers we've all grown so accustomed to.

• Forwards and backwards compatibility is not enforced: While forwards and backwards compatibility is a promise of Protobuf, actually maintaining backwards-compatible Protobuf APIs isn't widely practiced, and is hard to enforce.

• Stub distribution is a difficult, unsolved process: Organizations have to choose to either centralize their protoc workflow and distribute generated code, or require all service clients to run protoc independently. Because there is a steep learning curve to using protoc (and the associated protoc plugins) in a reliable manner, organizations often struggle with distributing their Protobuf files and stubs. This creates substantial overhead, and often requires a dedicated team to manage the process. Even when using a build system like Bazel, exposing APIs to external customers remains problematic.

• The tooling ecosystem is limited: Many user-friendly tools exist for REST/JSON APIs today. On the other hand, mock server generation, fuzz testing, documentation, and other daily API concerns are not widely standardized or user friendly for Protobuf APIs. As a result, teams regularly reinvent the wheel and build custom tooling to replicate the JSON ecosystem.

Buf is building a modern Protobuf ecosystem

Our tools address many of the problems above, ultimately allowing you to redirect much of your time and energy from managing Protobuf files to implementing your core features and infrastructure.

The Buf CLI

The Buf CLI enables you to create consistent Protobuf APIs that preserve compatibility and comply with best practices. The tool is currently available on an open-source basis. The Buf CLI incorporates these components to help you create consistent Protobuf APIs:

• A high-performance Protobuf compiler.
• A linter that enforces good API design choices and structure.
• A breaking change detector that enforces compatibility at the source code or wire level.
• A code generator that invokes your protoc plugins based on a configurable template.

The Buf Schema Registry (BSR)

The Buf Schema Registry (BSR) is a hosted SaaS platform that serves as your organization’s source of truth for your Protobuf APIs. It introduces dependency management to the Protobuf ecosystem, so that you can declare, resolve and use hosted BSR modules as dependencies, instead of vendoring .proto files across your organization. The BSR interacts directly with the Buf CLI to fetch your dependencies (analogous to npm for Node.js, pip for Python, cargo for Rust, and Go modules in Go).

The BSR also makes generating and consuming client code simpler with two additional features:

• Remote plugins: The BSR hosts commonly-used Protobuf plugins that you reference in a config file instead of requiring local installation, reducing the potential for API drift when local plugins get out of sync.

• Generated SDKs: The BSR generates SDKs from your .proto files when you push a module. This enables your API consumers to skip generating code altogether, and instead access generated code through tools they already know, like go get, npm install, mvn, gradle, or Swift Package Manager.

Where to go from here

See the installation page to install the buf CLI.

Next, we recommend completing the Buf CLI tour and the BSR tour. The tours provide an overview of most of Buf's existing functionality and take approximately 20 minutes to complete.

After completing the tour, check out the remainder of the documentation for your specific areas of interest. We've aimed to provide as much documentation as we can for the various components of Buf to give you a full understanding of Buf's surface area.

Finally, follow the project on GitHub, and contact us if you'd like to get involved.