Skip to content

Bufstream on MINIO AIStor#

MINIO AIStor delivers performance, scale, agility, and economics for large-scale workloads. AI Store is S3-compatible and runs anywhere from edge to core to cloud.

Prerequisites#

  • A Bufstream container image
  • MINIO AIStor up and running. If you don't have an AIStor license, you can request one here.
  • MINIO CLI
  • This should be a new Bufstream deployment. This guide does not cover migrating between storage platforms.

Configuring storage#

MINIO AIStor is S3-compatible and can be configured using the S3 storage provider in either bufstream.yaml or Helm values.yaml. MINIO AIStor specific values must be used for endpoint:

  • endpoint: Use the https:// endpoint of your MINIO AIStor deployment.
  • region: Use the region of your MINIO AIStor deployment. The default is us-east-1.

Continue reading for detailed instructions.

Create a MINIO AIStor bucket#

To get started with MINIO AIstor, create a new bucket using the MINIO CLI:

mc mb myaistor/mybucket

Copy its name into your notes: you'll need it for configuring Bufstream.

Configure Bufstream for MINIO AIStor#

To configure Bufstream to write to MINIO AIStor, you will need your endpoint, bucket name, access key, and secret access key. Your access key and secret access key values are the same as those you used to deploy MINIO AIStor.

Add or change Bufstream's data configuration in your bufstream.yaml file:

data:
  s3:
    uri: s3://<minio-bucket-name>/
    region: <minio-region-name>
    endpoint: <minio-https-endpoint>
    access_key_id:
      string: <minio-access-key-id>
    secret_access_key:
      string: <minio-secret-access-key>

Create a Kubernetes secret containing your MINIO Secret Access Key:

$ kubectl create secret --namespace bufstream generic bufstream-storage \
    --from-literal=secret_access_key=<minio-secret-access-key>

Then, add or change Bufstream's storage configuration in your values.yaml file:

 storage:
    use: s3
    s3:
        accessKeyId: "<minio-access-key-id>"
        secretName: bufstream-storage
        bucket: <minio-bucket-name>
        region: <minio-region-name>
        endpoint: "<minio-https-endpoint>"

Start Bufstream#

With MINIO AIStor configured for storage, start Bufstream. (For Kubernetes, follow the steps in the Bufstream deployment guide to deploy Bufstream or update the Helm chart.)

Use the Bufstream CLI's admin status command to verify a healthy deployment. You should see a status report like the following:

$ bufstream admin status
Name                         Status   Message
kafka_server                 healthy  Connected to localhost:9092
metadata_storage             healthy
metadata_storage_migrations  healthy  Migrations: current: 1, target: 1
object_storage               healthy
postgresql                   healthy  PostgreSQL: v14.18, connections: 20/100, connected apps: other: 19/25, uptime: 1m10s
version                      healthy  Version: <version>

A healthy status for object_storage means Bufstream has connected to MINIO and is ready to receive messages.

Testing MINIO AIStor (optional)#

You can use the Kafka client, GUI, operator, or CLI of your choice to test your MINIO integration.

Create a topic#

Create a topic. A kafkactl example is shown below:

$ kafkactl create topic bufstream-on-minio
topic created: bufstream-on-minio

Produce to the topic#

Use your preferred tool to produce a message. With kafkactl:

$ kafkactl produce bufstream-on-minio --value=hello-from-bufstream
message produced (partition=0   offset=0)

Open your MINIO AIStor console to the bucket you created and you'll see that Bufstream's added keys for topic data. Feel free to add more messages and topics, keeping an eye on the bucket for changes.

Wrapping up#

You've successfully produced data to a topic stored in MINIO AIStor. From here, you can rest easy knowing that MINIO securely backs up, recovers, and heals your data no matter where it's stored.