Skip to content

Bufstream on Tigris#

Tigris is a globally distributed, multi-cloud object storage platform with native S3 API support and zero egress fees. It dynamically places data in the region where it's being accessed—eliminating cross-cloud data transfer costs without sacrificing performance.

When you combine Bufstream and Tigris, you get unlimited message retention and truly global operation. Combining zero egress fees with typed streams means that your applications can scale across the globe fearlessly.

Prerequisites#

  • A Bufstream container image
  • A Tigris account; if you don't have one, you can create one at storage.new.
  • This should be a new Bufstream deployment. This guide does not cover migrating between storage platforms.

Configuring storage#

Tigris is S3-compatible and can be configured using the S3 storage provider in either bufstream.yaml or Helm values.yaml. Tigris-specific values must be used for region and endpoint:

  • region - Use auto to ensure that Tigris routes to the closest region.
  • endpoint - Use https://t3.storage.dev.

Continue reading for detailed instructions.

Create a Tigris bucket#

To get started with Tigris, create a new bucket at storage.new. Use the Standard access tier. Copy its name into your notes: you'll need it for configuring Bufstream.

Create a new access key with Editor permissions for the bucket. Make note of your Access Key ID and Secret Access Key: you'll need them for configuring Bufstream.

Configure Bufstream for Tigris#

Add or change Bufstream's storage configuration in your values.yaml file:

storage:
  provider: S3
  region: auto
  bucket: <bucket-name>
  endpoint: https://t3.storage.dev
  access_key_id:
    string: <tigris-access-key-id>
  secret_access_key:
    string: <tigris-secret-access-key>

Create a Kubernetes secret containing your Tigris Secret Access Key:

$ kubectl create secret --namespace bufstream generic bufstream-storage \
    --from-literal=secret_access_key=<tigris-secret-access-key>

Then, add or change Bufstream's storage configuration in your values.yaml file:

 storage:
    use: s3
    s3:
        accessKeyId: "AKIAIOSFODNN7EXAMPLE"
        secretName: bufstream-storage
        bucket: <bucket-name>
        region: auto
        endpoint: "https://t3.storage.dev"

Start Bufstream#

With Tigris configured for storage, start Bufstream. (For Kubernetes, follow the steps in the Bufstream deployment guide to deploy Bufstream or update the Helm chart.)

Use the Bufstream CLI's admin status command to verify a healthy deployment. You should see a status report like the following:

$ bufstream admin status
Name                         Status   Message
kafka_server                 healthy  Connected to localhost:9092
metadata_storage             healthy
metadata_storage_migrations  healthy  Migrations: current: 1, target: 1
object_storage               healthy
postgresql                   healthy  PostgreSQL: v14.18, connections: 20/100, connected apps: other: 19/25, uptime: 1m10s
version                      healthy  Version: <version>

A healthy status for object_storage means Bufstream has connected to Tigris and is ready to receive messages.

Testing Tigris (optional)#

You can use the Kafka client, GUI, operator, or CLI of your choice to test your Tigris integration.

Create a topic#

Create a topic. A kafkactl example is shown below:

$ kafkactl create topic bufstream-on-tigris
topic created: bufstream-on-tigris

Produce to the topic#

Use your preferred tool to produce a message. With kafkactl:

$ kafkactl produce bufstream-on-tigris --value=hello-from-bufstream
message produced (partition=0   offset=0)

Open your Tigris console to the bucket you created and you'll see that Bufstream's added keys for topic data. Feel free to add more messages and topics, keeping an eye on the bucket for changes.

Wrapping up#

You've successfully produced data to a topic stored in Tigris. From here, you can rest easy knowing that Tigris securely backs up your data, and you can access it from anywhere in the world.

To learn more about how Bufstream and Tigris work together, check out the this blog post from Tigris.