Bufstream

Bufstream / Reference / Bufstream configuration files

Bufstream Helm values

The Bufstream Helm chart is used to deploy Bufstream on Kubernetes. The Bufstream Helm values.yaml file defines the configuration for the Bufstream Kubernetes cluster. The configuration parameters and defaults documented below are the common and recommended settings for deploying a Bufstream cluster.

Values and defaults

`bufstream.deployment.affinity`

object

Bufstream Deployment Affinity.

Defaults to {}.

`bufstream.deployment.args`

list

Bufstream Deployment args to be appended.

Defaults to [].

`bufstream.deployment.autoscaling.behavior`

object

Horizontal Pod Autoscaler behavior.

Defaults to {}.

`bufstream.deployment.autoscaling.enabled`

bool

Whether to enable the horizontal pod autoscaler.

Defaults to false.

`bufstream.deployment.autoscaling.maxReplicas`

int

Maximum number of autoscaler allowed replicas.

Defaults to 18.

`bufstream.deployment.autoscaling.minReplicas`

int

Minimum number of autoscaler allowed replicas.

Defaults to 6.

`bufstream.deployment.autoscaling.targetCPU`

string

Target CPU threshold for managing replica count.

Defaults to "50".

`bufstream.deployment.autoscaling.targetMemory`

string

Target memory threshold for managing replica count.

Defaults to "".

`bufstream.deployment.command`

list

Bufstream Deployment command.

Defaults to ["/usr/local/bin/bufstream"].

`bufstream.deployment.extraContainerPorts`

object

Bufstream Deployment Extra container ports for the bufstream container.

Defaults to {}.

`bufstream.deployment.extraContainers`

list

Bufstream Deployment additional containers to run besides the bufstream container.

Defaults to [].

`bufstream.deployment.extraEnv`

list

Bufstream Deployment Extra environment variables for the bufstream container.

Defaults to [].

`bufstream.deployment.extraVolumeMounts`

list

Bufstream Deployment Extra volume mounts for the bufstream container.

Defaults to [].

`bufstream.deployment.extraVolumes`

list

Bufstream Deployment Extra volumes.

Defaults to [].

`bufstream.deployment.livenessProbe.failureThreshold`

int

Bufstream Deployment Liveness Probe Maximum failure threshold.

Defaults to 3.

`bufstream.deployment.nodeSelector`

object

Bufstream Deployment Node selector.

Defaults to {}.

`bufstream.deployment.podAnnotations`

object

Bufstream Deployment Pod annotations.

Defaults to {}.

`bufstream.deployment.podLabels`

object

Bufstream Deployment Pod labels.

Defaults to {}.

`bufstream.deployment.replicaCount`

int

Bufstream Deployment replica count.

Defaults to 3.

`bufstream.deployment.resources.limits.cpu`

string

Bufstream Deployment Resource request CPU.

Defaults to "".

`bufstream.deployment.resources.limits.memory`

string

Bufstream Deployment Resource limits memory.

Defaults to "8Gi".

`bufstream.deployment.resources.requests.cpu`

int

Bufstream Deployment Resource request CPU.

Defaults to 2.

`bufstream.deployment.resources.requests.memory`

string

Bufstream Deployment Resource request memory.

Defaults to "8Gi".

`bufstream.deployment.shareProcessNamespace`

bool

Bufstream Deployment setting for sharing the process namespace.

Defaults to false.

`bufstream.deployment.startupProbe.failureThreshold`

int

Bufstream Deployment Liveness Probe Configuration

Defaults to 3.

`bufstream.deployment.terminationGracePeriodSeconds`

int

Bufstream Deployment termination grace period.

Defaults to 420.

`bufstream.deployment.tolerations`

list

Bufstream Deployment Tolerations.

Defaults to [].

`bufstream.image.pullPolicy`

string

Bufstream Deployment container image pull policy.

Defaults to "IfNotPresent".

`bufstream.image.repository`

string

Bufstream Deployment container image repository.

Defaults to "us-docker.pkg.dev/buf-images-1/bufstream-public/images/bufstream".

`bufstream.image.tag`

string

Overrides the image tag whose default is the chart version.

Defaults to "latest".

`bufstream.podDisruptionBudget.enabled`

bool

Whether to enable pod disruption budget.

Defaults to false.

`bufstream.podDisruptionBudget.maxUnavailable`

string

Number of pods that are unavailable after eviction as number or percentage (eg.: 50%). Has higher precedence over minAvailable

Defaults to "".

`bufstream.podDisruptionBudget.minAvailable`

string

Number of pods that are available after eviction as number or percentage (eg.: 50%).

Defaults to "" (defaults to 0 if not specified).

`bufstream.service.annotations`

object

Kubernetes Service annotations.

Defaults to {}.

`bufstream.service.enabled`

bool

Whether to create a Kubernetes Service for this bufstream deployment.

Defaults to true.

`bufstream.service.type`

string

Kubernetes Service type.

Defaults to "ClusterIP".

`bufstream.serviceAccount.annotations`

object

Kubernetes Service Account annotations.

Defaults to {}.

`bufstream.serviceAccount.create`

bool

Whether to create a Kubernetes Service Account for this bufstream deployment.

Defaults to true.

`bufstream.serviceAccount.name`

string

Kubernetes Service Account name.

Defaults to "bufstream-service-account".

`cluster`

string

The name of the cluster. Used by bufstream to identify itself.

Defaults to "bufstream".

`configOverrides`

object

Bufsteam configuration overrides. Any value here will be set directly on the bufstream config.yaml, taking precedence over any other helm defined values.

Defaults to {}.

`dataEnforcement`

object

Configuration for data enforcement via schemas of records flowing in and out of the agent.

Defaults to {}.

`extraObjects`

list

Extra Kubernetes objects to install as part of this chart.

Defaults to [].

`imagePullSecrets`

list

Reference to one or more secrets to be used when pulling images. For more information, see Pull an Image from a Private Registry.

Defaults to [].

`kafka.address`

object

The address the Kafka server should listen on. This defaults to the K8S pod IP and 9092 port.

Defaults to {host: "<pod ip>", port: 9092}.

`kafka.exactLogOffsets`

bool

If exact log hwm and start offsets should be computed when fetching records.

Defaults to false.

`kafka.exactLogSizes`

bool

If exact log sizes should be fetched when listing sizes for all topics/partitions.

Defaults to true.

`kafka.fetchEager`

bool

If a fetch should return as soon as any records are available.

Defaults to true.

`kafka.fetchSync`

bool

If fetches from different readers should be synchronized to improve cache hit rates.

Defaults to true.

`kafka.groupConsumerSessionTimeout`

string

The default group consumer session timeout.

Defaults to "45s".

`kafka.groupConsumerSessionTimeoutMax`

string

The maximum group consumer session timeout.

Defaults to "60s".

`kafka.groupConsumerSessionTimeoutMin`

string

The minimum group consumer session timeout.

Defaults to "10s".

`kafka.idleTimeout`

int

How long a Kafka connection can be idle before being closed by the server. If set a value less than or equal to zero, the timeout will be disabled.

Defaults to 0.

`kafka.numPartitions`

int

The default number of partitions to use for a new topic.

Defaults to 1.

`kafka.partitionBalanceStrategy`

string

How to balance topic/partitions across bufstream nodes. One of: ["BALANCE_STRATEGY_UNSPECIFIED", "BALANCE_STRATEGY_PARTITION", "BALANCE_STRATEGY_HOST", "BALANCE_STRATEGY_CLIENT_ID"]

Defaults to "BALANCE_STRATEGY_PARTITION".

`kafka.produceConcurrent`

bool

If records from a producer to different topic/partitions may be sequenced concurrently instead of serially.

Defaults to true.

`kafka.publicAddress`

object

The public address clients should use to connect to the Kafka server, if different from address.

Defaults to {}.

`kafka.requestBufferSize`

int

The number of kafka request to unmarshal and buffer before processing.

Defaults to 5.

`kafka.tlsCertificateSecrets`

list

Kubernetes secrets containing a tls.crt and tls.key (as the secret keys, see https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets) to present to the client. The first certificate compatible with the client's requirements is selected automatically.

Defaults to [].

`kafka.tlsClientAuth`

string

Declare the policy the server will follow for mutual TLS (mTLS). Supported values are [NO_CERT, REQUEST_CERT, REQUIRE_CERT, VERIFY_CERT_IF_GIVEN, REQUIRE_AND_VERIFY_CERT]. Only supported when using tlsCertificateSecret.

Defaults to "NO_CERT".

`kafka.tlsClientCasSecret`

string

Kubernetes seccret containing a tls.crt (as the secret key) PEM-encoded certificate authorities used by the server to validate the client certificates. This field cannot be empty if tlsClientAuth is set for client performing verification. Only supported when using tlsCertificateSecret.

Defaults to "".

`kafka.zoneBalanceStrategy`

string

How to balance clients across zones, when client does not specify a zone. One of: ["BALANCE_STRATEGY_UNSPECIFIED", "BALANCE_STRATEGY_PARTITION", "BALANCE_STRATEGY_HOST", "BALANCE_STRATEGY_CLIENT_ID"]

Defaults to "BALANCE_STRATEGY_PARTITION".

`metadata.etcd.addresses`

list

Etcd addresses to connect to.

Defaults to [].

`metadata.use`

string

Which metadata storage that bufstream is using. Currently, only etcd is supported.

Defaults to "etcd".

`nameOverride`

string

Overrides .Chart.Name throughout the chart.

Defaults to "".

`namespaceCreate`

bool

Whether to create the namespace where resources are located.

Defaults to false.

`namespaceOverride`

string

Will be used as the namespace for all resources instead of .Release.namespace if set

Defaults to "".

`observability.logLevel`

string

Log level to use.

Defaults to "INFO".

`observability.metrics.exporter`

string

Open Telemetry exporter. Supports [NONE, STDOUT, HTTP, HTTPS, PROMETHEUS].

Defaults to "NONE".

`observability.otlpEndpoint`

string

Open Telemetry endpoint to push metrics.

Defaults to "".

`observability.tracing.exporter`

string

Open Telemetry exporter. Supports [NONE, STDOUT, HTTP, HTTPS, PROMETHEUS].

Defaults to "NONE".

`observability.tracing.traceRatio`

float

Trace sample ratio.

Defaults to 0.1.

`storage.gcs.bucket`

string

GCS bucket name.

Defaults to "".

`storage.gcs.prefix`

string

GCS prefix to use for all stored files.

Defaults to "".

`storage.gcs.secretName`

string

Kubernetes secret containing a credentials.json (as the secret key) service account key to use instead of the metadata server.

Defaults to "".

`storage.s3.accessKeyId`

string

S3 Access Key ID to use instead of the metadata server.

Defaults to "".

`storage.s3.bucket`

string

S3 bucket name.

Defaults to "".

`storage.s3.forcePathStyle`

bool

S3 Force Path Style setting. See https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-s3.html.

Defaults to false.

`storage.s3.prefix`

string

S3 prefix to use for all stored files.

Defaults to "".

`storage.s3.region`

string

S3 bucket region.

Defaults to "".

`storage.s3.secretName`

string

Kubernetes secret containing a secret_access_key (as the secret key) to use instead of the metadata server.

Defaults to "".

`storage.use`

string

Which object storage that bufstream is using. Currently, gcs and s3 are supported.

Defaults to "s3".

`zone`

string

The zone location of brokers, e.g., the datacenter/availability zone where the agent is running. If not given, bustream will try to infer this from node metadata. This is currently for bufstream internal functionality, and does not control cloud providers such as GCP directly.

Defaults to "".

Annotated `values.yaml`

The Helm values.yaml file below contains all of the configuration parameters for a Bufstream cluster with recomemnded defaults. You can copy this annotated YAML into your Helm values file to use as a reference when configuring and deploying Bufstream.

# -- Overrides .Chart.Name throughout the chart.
nameOverride: ""
# -- Will be used as the namespace for all resources instead of .Release.namespace if set
namespaceOverride: ""
# -- Whether to create the namespace where resources are located.
namespaceCreate: false
# -- Reference to one or more secrets to be used when pulling images.
# For more information, see [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
imagePullSecrets: []
# -- The name of the cluster. Used by bufstream to identify itself.
cluster: bufstream
# -- The zone location of brokers, e.g., the datacenter/availability zone where the agent is running.
# If not given, bustream will try to infer this from node metadata.
# This is currently for bufstream internal functionality, and does not control cloud providers such as GCP directly.
zone: ""
# -- Configuration for data enforcement via schemas of records flowing in and out of the agent.
dataEnforcement: {}
kafka:
  # -- The address the Kafka server should listen on. This defaults to the K8S pod IP and 9092 port.
  # @default -- `{host: "<pod ip>", port: 9092}`
  address: {}
  # -- The public address clients should use to connect to the Kafka server, if different from `address`.
  publicAddress: {}
  # -- Kubernetes secrets containing a `tls.crt` and `tls.key` (as the secret keys, see https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets) to present to the client. The first certificate compatible with the client's requirements is selected automatically.
  tlsCertificateSecrets: []
  # -- Declare the policy the server will follow for mutual TLS (mTLS). Supported values are [NO_CERT, REQUEST_CERT, REQUIRE_CERT, VERIFY_CERT_IF_GIVEN, REQUIRE_AND_VERIFY_CERT]. Only supported when using tlsCertificateSecret.
  # @default -- `"NO_CERT"`
  tlsClientAuth: ""
  # -- Kubernetes seccret containing a `tls.crt` (as the secret key) PEM-encoded certificate authorities used by the server to validate the client certificates. This field cannot be empty if tlsClientAuth is set for client performing verification. Only supported when using tlsCertificateSecret.
  tlsClientCasSecret: ""
  # -- If a fetch should return as soon as any records are available.
  fetchEager: true
  # -- If fetches from different readers should be synchronized to improve cache hit rates.
  fetchSync: true
  # -- If records from a producer to different topic/partitions may be sequenced concurrently instead of serially.
  produceConcurrent: true
  # -- How to balance clients across zones, when client does not specify a zone. One of: ["BALANCE_STRATEGY_UNSPECIFIED", "BALANCE_STRATEGY_PARTITION", "BALANCE_STRATEGY_HOST", "BALANCE_STRATEGY_CLIENT_ID"]
  zoneBalanceStrategy: BALANCE_STRATEGY_PARTITION
  # -- How to balance topic/partitions across bufstream nodes. One of: ["BALANCE_STRATEGY_UNSPECIFIED", "BALANCE_STRATEGY_PARTITION", "BALANCE_STRATEGY_HOST", "BALANCE_STRATEGY_CLIENT_ID"]
  partitionBalanceStrategy: BALANCE_STRATEGY_PARTITION
  # -- The number of kafka request to unmarshal and buffer before processing.
  requestBufferSize: 5
  # -- How long a Kafka connection can be idle before being closed by the server. If set a value less than or equal to zero, the timeout will be disabled.
  idleTimeout: 0
  # -- The default number of partitions to use for a new topic.
  numPartitions: 1
  # -- If exact log sizes should be fetched when listing sizes for all topics/partitions.
  exactLogSizes: true
  # -- If exact log hwm and start offsets should be computed when fetching records.
  exactLogOffsets: false
  # -- The default group consumer session timeout.
  groupConsumerSessionTimeout: 45s
  # -- The minimum group consumer session timeout.
  groupConsumerSessionTimeoutMin: 10s
  # -- The maximum group consumer session timeout.
  groupConsumerSessionTimeoutMax: 60s

metadata:
  # -- Which metadata storage that bufstream is using.
  # Currently, only `etcd` is supported.
  use: etcd
  etcd:
    # -- Etcd addresses to connect to.
    addresses: []
    # addresses:
    # - host: ""
    #   port: 2379
storage:
  # -- Which object storage that bufstream is using.
  # Currently, `gcs` and `s3` are supported.
  use: s3
  gcs:
    # -- GCS bucket name.
    bucket: ""
    # -- GCS prefix to use for all stored files.
    prefix: ""
    # -- Kubernetes secret containing a `credentials.json` (as the secret key) service account key to use instead of the metadata server.
    secretName: ""
  s3:
    # -- S3 bucket name.
    bucket: ""
    # -- S3 bucket region.
    region: ""
    # -- S3 prefix to use for all stored files.
    prefix: ""
    # -- S3 Force Path Style setting. See https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-s3.html.
    forcePathStyle: false
    # -- S3 Access Key ID to use instead of the metadata server.
    accessKeyId: ""
    # -- Kubernetes secret containing a `secret_access_key` (as the secret key) to use instead of the metadata server.
    secretName: ""
observability:
  # -- Log level to use.
  logLevel: INFO
  # -- Open Telemetry endpoint to push metrics.
  otlpEndpoint: ""
  metrics:
    # -- Open Telemetry exporter. Supports [NONE, STDOUT, HTTP, HTTPS, PROMETHEUS].
    exporter: "NONE"
  tracing:
    # -- Open Telemetry exporter. Supports [NONE, STDOUT, HTTP, HTTPS, PROMETHEUS].
    exporter: "NONE"
    # -- Trace sample ratio.
    traceRatio: 0.1
bufstream:
  service:
    # -- Whether to create a Kubernetes Service for this bufstream deployment.
    enabled: true
    # -- Kubernetes Service type.
    type: ClusterIP
    # -- Kubernetes Service annotations.
    annotations: {}
  serviceAccount:
    # -- Whether to create a Kubernetes Service Account for this bufstream deployment.
    create: true
    # -- Kubernetes Service Account name.
    name: bufstream-service-account
    # -- Kubernetes Service Account annotations.
    annotations: {}
  deployment:
    # -- Bufstream Deployment command.
    # @default -- `["/usr/local/bin/bufstream"]`
    command: []
    # -- Bufstream Deployment args to be appended.
    args: []
    # -- Bufstream Deployment replica count.
    replicaCount: 3
    # -- Bufstream Deployment Pod annotations.
    podAnnotations: {}
    # -- Bufstream Deployment Pod labels.
    podLabels: {}
    resources:
      requests:
        # -- Bufstream Deployment Resource request CPU.
        cpu: 2
        # -- Bufstream Deployment Resource request memory.
        memory: 8Gi
      limits:
        # -- Bufstream Deployment Resource request CPU.
        cpu: ""
        # -- Bufstream Deployment Resource limits memory.
        memory: 8Gi
    # -- Bufstream Deployment Node selector.
    nodeSelector: {}
    # -- Bufstream Deployment Affinity.
    affinity: {}
    # -- Bufstream Deployment Tolerations.
    tolerations: []
    # -- Bufstream Deployment Extra environment variables for the bufstream container.
    extraEnv: []
    # -- Bufstream Deployment Extra volume mounts for the bufstream container.
    extraVolumeMounts: []
    # -- Bufstream Deployment Extra volumes.
    extraVolumes: []
    # -- Bufstream Deployment Extra container ports for the bufstream container.
    extraContainerPorts: {}
    # -- Bufstream Deployment additional containers to run besides the bufstream container.
    extraContainers: []
    livenessProbe:
      # -- Bufstream Deployment Liveness Probe Maximum failure threshold.
      failureThreshold: 3
    startupProbe:
      # -- Bufstream Deployment Liveness Probe Configuration
      failureThreshold: 3
    # -- Bufstream Deployment termination grace period.
    terminationGracePeriodSeconds: 420 # 7 minutes
    # -- Bufstream Deployment setting for sharing the process namespace.
    shareProcessNamespace: false
    autoscaling:
      # -- Whether to enable the horizontal pod autoscaler.
      enabled: false
      # -- Minimum number of autoscaler allowed replicas.
      minReplicas: 6
      # -- Maximum number of autoscaler allowed replicas.
      maxReplicas: 18
      # -- Target CPU threshold for managing replica count.
      targetCPU: "50"
      # -- Target memory threshold for managing replica count.
      targetMemory: ""
      # -- [Horizontal Pod Autoscaler behavior.](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior)
      behavior: {}
  podDisruptionBudget:
    # -- Whether to enable pod disruption budget.
    enabled: false
    # -- Number of pods that are available after eviction as number or percentage (eg.: 50%).
    # @default -- `""` (defaults to 0 if not specified)
    minAvailable: ""
    # -- Number of pods that are unavailable after eviction as number or percentage (eg.: 50%). Has higher precedence over `minAvailable`
    maxUnavailable: ""
  image:
    # -- Bufstream Deployment container image repository.
    repository: us-docker.pkg.dev/buf-images-1/bufstream-public/images/bufstream
    # -- Overrides the image tag whose default is the chart version.
    tag: "latest"
    # -- Bufstream Deployment container image pull policy.
    pullPolicy: IfNotPresent
# -- Extra Kubernetes objects to install as part of this chart.
extraObjects: []
# -- Bufsteam configuration overrides. Any value here will be set directly on the bufstream config.yaml, taking precedence over any other helm defined values.
configOverrides: {}

To configure Bufstream with recommended, cloud-specific settings, consult the AWS and GCP documentation.