Sign inSign up
Buf Schema Registry (BSR)
Buf Schema Registry (BSR) / Private BSR / On-prem

Observability

Observability is the ability to measure the internal states of a system by examining its outputs. Monitoring incoming calls, error scenarios, and traces is critical to ensure a robust system is running properly.

The rules and dashboards can be downloaded as a single zip file from the Release notes section below.

Promethues Rules

Buf provides a set of prometheus alerting rules and recording rules for monitoring the health of your BSR instance.

  • prometheus-slo-rpc-rules.yaml provides alerting rules and recording rules for all RPC endpoints.
  • prometheus-slo-http-rules.yaml provides alerting rules and recording rules for all HTTP endpoints.
  • prometheus-slo-registry-rules.yaml provides alerting rules and recording rules for language registry endpoints that serve Generated SDKs.

Grafana Dashboards

To import the dashboards, follow Granafa's instructions. The Prometheus recording rules are required to use the dashboards, since their precomputed metrics are necessary to establish the SLO window on the dashboards.

  • grafana-slo-rpc-dashboard.json is for monitoring all RPC traffic with the capability to group by cluster, rpc_service, and/or rpc_method.
  • grafana-slo-http-dashboard.json is for monitoring all HTTP traffic with the capability to group by cluster.
  • grafana-slo-registry-dashboard.json is for monitoring language registry endpoints that serve Generated SDKs.

Dashboards

The dashboards included with the BSR expose its overall health and can aid in identifying and diagnosing operational issues. Several charts and components on the dashboards are described in more detail below.

Dashboard

Service Level Objectives

Each dashboard is primarily defined in terms of service level objectives (SLOs).

Each dashboard has a success rate and latency objective, and the dashboards keeps track of when those objectives are met or missed. The success rate objective is currently 99.5% for all dashboards, and the latency objectives can be viewed in the config panel of the dashboards. Generally, the dashboards consider a request failed if it returns an unsuccessful error code (e.g. 5xx or the Connect RPC equivalent) or if it exceeds its latency target.

Filters

Filters can be set to restrict the set of data the dashboard operates on. Setting cluster is important for sites with multiple BSR clusters. Other filters can be set based on which SLO dashboard is being used:

  • slo-http: cluster
  • slo-rpc: cluster, rpc_method, and rpc_service
  • slo-registry: cluster and request_type

High Level Information

The first two rows of the dashboard contain high level information to quickly provide the current operational status of the BSR.

First Row Boxes

The Availability, Error Budget, Slow Requests, Failed Requests, and Requests boxes display aggregate counts that give a very high level overview of what is happening with the BSR over the trailing SLO evaluation window.

SLO Table

This panel lists each grouping that has been observed and its availability. The lowest availability pairs are listed first to call out any parts of the BSR that may be having issues.

Error Budget Percentage

This panel shows the aggregate (based on filters) error budget burn down for the BSR. Each point on this chart shows the error budget used up over the trailing SLO evaluation window.

Other Dashboard Components

Most of the other dashboard panels have titles that explain exactly what they are displaying and should be self-explanatory, others are clarified below.

Firing Alerts

Any alerts that are firing will be visible in this panel.

Config

Internal configuration for the dashboard. Currently cannot be changed by end users.

Alerts Overview

Alerts are configured for every grouping on the dashboard in each deployed BSR cluster. Each grouping has an error budget based on an availability objective of 99.5% over the previous four weeks. An alert will fire if a given method returns errors at a rate that threatens the 99.5% objective. There are two classes of alerts defined by how rapidly they notify of errors.

High Priority Alerts respond swiftly, signalling immediate threats to the system. For these alerts to fire, 50% of the error budget must be consumed in the last hour and the method needs to have received at least 10 requests in that period.

Low Priority Alerts are designed for longer durations, capturing potential issues without responding to minor fluctuations. For these alerts to fire, 10% of the error budget must be consumed in the last 24 hours and the method needs to have had at least 10 requests in that period.

Simply put, if errors accumulate too quickly within the past one hour or 24 hours, alerts are triggered.

Diagnosing Alerts

When an alert fires, it's accompanied by specific labels to aid in your investigation:cluster, rpc_method, rpc_service and request_type , depending on which dashboard is being viewed. These labels highlight the affected areas, helping you pinpoint the problem's origin.

Once an alert is triggered:

  1. Firing Alerts Panel: Here, you can view all active alerts, including the one that notified you.
  2. Error Rate Diagram: Look for spikes in this chart, giving you a visual representation of when and where issues arose.
  3. Drill Down: Utilize the provided labels (cluster, rpc_method, request_type and rpc_service) to refine your search in the dashboard and focus on the affected areas.

By following these steps and utilizing the dashboard, you can swiftly identify, understand, and address any issues in your system.

Release Notes

v1.3.1

Release Date: 2024-03-19 | Minimum BSR version: v1.3.0

  • The observability dashboards and alerts have been updated to fix a bug where an incorrect histogram bucket was being used.

Download v1.3.1 rules and dashboards

Expand to see previous versions

v1.3.0

Release Date: 2023-12-05 | Minimum BSR version: v1.3.0

  • The SLO alerting threshold for slow requests has been decreased from 250s to 75s, in an effort to be more responsive to slow requests. Additionally, the histogram buckets for HTTP metrics have been adjusted so they are now in parity with RPC metrics.

Download v1.3.0 rules and dashboards

v1.2.0

Release Date: 2023-10-03 | Minimum BSR version: v1.1.0

  • This release includes new SLO Dashboards for non-RPC endpoints to give visibility into the entire BSR as a whole. There are now separate dashboards to monitor all HTTP requests and Registry requests.

Download v1.2.0 rules and dashboards

v1.1.0

Release Date: 2023-08-28 | Minimum BSR version: v1.0.4

  • Enabled alerting for SLO dashboard
    • Adds high-priority  alerts with severity=page label for high error rate incidents ErrorBudgetBurn
    • Adds low-priority ErrorBudgetBurn alerts with severity=warning label for low error rate incidents

Download v1.1.0 rules and dashboards

v1.0.0

Release Date: 2023-07-14 | Minimum BSR version: v1.0.2

  • Initial version of Grafana dashboards and Prometheus recording rules

Download v1.0.0 rules and dashboards

  1. Promethues Rules
  2. Grafana Dashboards
  3. Dashboards
    1. Service Level Objectives
    2. Filters
    3. High Level Information
      1. First Row Boxes
      2. SLO Table
      3. Error Budget Percentage
    4. Other Dashboard Components
      1. Firing Alerts
      2. Config
  4. Alerts Overview
    1. Diagnosing Alerts
    2. Navigating the Dashboard
  5. Release Notes