Skip to content

Observability – Overview#

Bufstream is instrumented for monitoring cluster health, performance, and errors. Metrics are instrumented using OpenTelemetry, with support for standard exporters like OTLP and Prometheus. Specific configuration recommendations including dashboards and monitors/alerts for Datadog and Grafana are available.

Configuring metrics#

OTLP#

To configure Bufstream to report to an OpenTelemetry collector, configure Bufstream as follows:

metrics:
  otlp:
    # Required: Type of transport to use for OTLP. Must be "http" or "grpc".
    type: http
    # Required: URL of OTLP endpoint to export metrics to.
    url: <string>

observability:
  exporter:
    address: "<collector-hostname>:4318"
    # Set to true to report over HTTP, false to report over TLS.
    insecure: true
  metrics:
    exporterType: "OTLP_HTTP"

Prometheus#

Prometheus metrics are exposed at the debug address. To enable the Prometheus endpoint for collecting metrics, configure Bufstream as follows:

debug:
  # Address to listen for connections for debug information. If configured,
  # pprof and Prometheus exported metrics will be exposed on this address.
  listen_address: <hostport>

observability:
  metrics:
    exporterType: "PROMETHEUS"

The Prometheus endpoint (/metrics) will be available on the Bufstream broker on port 9090.

Logging#

Logging is output to standard error in the Bufstream brokers in JSON format. Use the following to configure the log level:

logging:
  # Log level, defaults to info.
  level: debug|info|warn|error

observability:
  # One of "DEBUG", "INFO", "WARN", or "ERROR".
  logLevel: "INFO"

Tracing#

Use the following to configure tracing:

traces:
  # Configuration for exporting OpenTelemetry-based traces.
  otlp:
    # Required: Type of transport to use for OTLP. Must be "http" or "grpc".
    type: http
    # Required: URL of OTLP endpoint to export traces to.
    url: <string>
    # OpenTelemetry trace sample ratio, defaults to 0.1.
    trace_ratio: 0.1

observability:
  exporter:
    address: "<collector-hostname>:4318"
    # Set to true to report over HTTP, false to report over TLS.
    insecure: true
  tracing:
    exporterType: "OTLP_HTTP"
    # Optional, trace sampling ratio, defaults to 0.1
    # traceRatio: 0.1