Bufstream on MINIO AIStor#
MINIO AIStor delivers performance, scale, agility, and economics for large-scale workloads. AI Store is S3-compatible and runs anywhere from edge to core to cloud.
Prerequisites#
- A Bufstream container image
- MINIO AIStor up and running. If you don't have an AIStor license, you can request one here.
- MINIO CLI
- This should be a new Bufstream deployment. This guide does not cover migrating between storage platforms.
Configuring storage#
MINIO AIStor is S3-compatible and can be configured using the S3
storage provider in either bufstream.yaml or Helm values.yaml. MINIO AIStor specific values must be used for endpoint
:
endpoint
: Use thehttps://
endpoint of your MINIO AIStor deployment.region
: Use the region of your MINIO AIStor deployment. The default isus-east-1
.
Continue reading for detailed instructions.
Create a MINIO AIStor bucket#
To get started with MINIO AIstor, create a new bucket using the MINIO CLI:
Copy its name into your notes: you'll need it for configuring Bufstream.
Configure Bufstream for MINIO AIStor#
To configure Bufstream to write to MINIO AIStor, you will need your endpoint, bucket name, access key, and secret access key. Your access key and secret access key values are the same as those you used to deploy MINIO AIStor.
Add or change Bufstream's data
configuration in your bufstream.yaml
file:
Start Bufstream#
With MINIO AIStor configured for storage, start Bufstream. (For Kubernetes, follow the steps in the Bufstream deployment guide to deploy Bufstream or update the Helm chart.)
Use the Bufstream CLI's admin status command to verify a healthy deployment. You should see a status report like the following:
$ bufstream admin status
Name Status Message
kafka_server healthy Connected to localhost:9092
metadata_storage healthy
metadata_storage_migrations healthy Migrations: current: 1, target: 1
object_storage healthy
postgresql healthy PostgreSQL: v14.18, connections: 20/100, connected apps: other: 19/25, uptime: 1m10s
version healthy Version: <version>
A healthy
status for object_storage
means Bufstream has connected to MINIO and is ready to receive messages.
Testing MINIO AIStor (optional)#
You can use the Kafka client, GUI, operator, or CLI of your choice to test your MINIO integration.
Create a topic#
Create a topic. A kafkactl example is shown below:
Produce to the topic#
Use your preferred tool to produce a message. With kafkactl
:
$ kafkactl produce bufstream-on-minio --value=hello-from-bufstream
message produced (partition=0 offset=0)
Open your MINIO AIStor console to the bucket you created and you'll see that Bufstream's added keys for topic data. Feel free to add more messages and topics, keeping an eye on the bucket for changes.
Wrapping up#
You've successfully produced data to a topic stored in MINIO AIStor. From here, you can rest easy knowing that MINIO securely backs up, recovers, and heals your data no matter where it's stored.