Benchmarks and cost#
Bufstream is a Kafka-compatible message queue with built-in support for data governance and quality enforcement. But Bufstream isn't just adding features to Apache Kafka—built for the cloud, Bufstream is dramatically more cost-efficient to operate.
As shown in the following OpenMessaging Benchmark Framework results, Bufstream is 8x less expensive than Apache Kafka while remaining fully compatible with existing Kafka clients, tools, and applications.
Key takeaways#
- Bufstream kept up easily with industry-standard benchmark workloads, maintaining steady throughput with headroom for load spikes and opt-in features.
- Bufstream delivers 8x cost savings over Apache Kafka for high-throughput workloads.
- A latency trade-off exists but is reasonable: 260ms median, 500ms p99 vs. Kafka's lower latencies.
- Full Kafka protocol compatibility: Bufstream works with existing clients, connectors, and tools.
- Operational complexity is reduced through cloud-native architecture with automatic scaling and no disk management.
Benchmark specifications#
Benchmarking demands an apples-to-apples comparison, so we chose an industry-standard scenario: a single topic with 288 partitions and 1 GiB/s of symmetric reads and writes. While this scenario doesn't match every real-world workload, Bufstream handles a wide variety of workloads with ease.
This benchmark used the following workload and infrastructure:
- OpenMessaging Benchmark Framework used for standardized testing
- Single topic with 288 partitions
- 1 GiB/s symmetric read/write throughput
- 1,048,574 (2^20) messages per second
- 1 KiB partially randomized messages to simulate real-world compressibility
- 64 producers and 64 consumers
- 10-hour sustained test duration
- 6 Bufstream brokers
- AWS m6in.xlarge instances (4 vCPUs, 16 GiB memory each)
- 3 availability zones (2 brokers per zone)
- 1 standard S3 bucket
- 3 etcd nodes
- AWS m6in.large instances (2 vCPUs, 8 GiB memory each)
- One etcd node per zone for high availability
The OpenMessaging Benchmark Framework workload configuration used:
name: bufstream
topics: 1
partitionsPerTopic: 288
messageSize: 1024 # 1 KiB
producerRate: 1048576 # 2^20
producersPerTopic: 64
useRandomizedPayloads: true
randomBytesRatio: 0.25
randomizedPayloadPoolSize: 1024
subscriptionsPerTopic: 1
consumerPerSubscription: 64
consumerBacklogSizeGB: 0
testDurationMinutes: 600
warmupDurationMinutes: 5
Like most high-throughput benchmarks, we configured the Kafka producers and consumers to send fewer, larger batches:
name: bufstream
driverClass: io.openmessaging.benchmark.driver.kafka.KafkaBenchmarkDriver
replicationFactor: 3
reset: false
commonConfig: |
client.id=zone=${BUFSTREAM_ZONE_HINT},${HOSTNAME}
bootstrap.servers=bufstream:9092
request.timeout.ms=120000
producerConfig: |
linger.ms=100
# 1 MiB
batch.size=1048576
# 128 MiB
buffer.memory=134217728
# 64 MiB
max.request.size=67108864
compression.type=lz4
consumerConfig: |
auto.offset.reset=earliest
enable.auto.commit=true
# 100 MiB
max.partition.fetch.bytes=104857600
fetch.max.bytes=104857600
Clients were configured with a zone-specific ID, allowing Bufstream to minimize inter-zone networking. Off-the-shelf Kafka clients support setting the client ID, but we did have to patch the OpenMessaging Benchmark Framework to allow this configuration.
Predictable performance#
Benchmark results show Bufstream maintains steady throughput while operating well within resource limits, providing confidence that production deployments have enough headroom to handle load spikes, poorly configured clients, and opt-in features like semantic validation.
Write throughput#
The cluster kept up easily, sustaining write throughput over 1 GiB/s uncompressed and 256 MiB/s compressed (a 4:1 compression ratio).
CPU and memory usage#
Each Bufstream broker used less than 1 of the 4 vCPUs available. Memory usage—mostly driven by an in-memory cache—was less than 75%.
Latency characteristics#
There's no free lunch: object storage is durable, scalable, and cost-efficient, but it's not as low-latency as local disk. The benchmark used Bufstream's default configuration, striking a balance between end-to-end message delivery latency and cost. Median end-to-end latency was 260 milliseconds and p99 latency was 500 milliseconds.
For the high-volume analytic workloads that drive most Kafka usage, this latency is more than acceptable—especially paired with dramatic cost savings.
Metadata usage#
The benchmark used etcd as a distributed metadata store for recently written messages. The cluster remained lightly loaded, with each etcd node using less than 0.25 vCPUs and 150 MiB of memory.
Because metadata is moved out of etcd and into object storage within a few minutes, the etcd total database size stayed constant at 35 MiB.
Benchmark costs#
The Bufstream cluster used in the benchmark costs just $11,147 per month.
Optimized for cloud economics#
Bufstream's architecture specifically leverages cloud economics, using object storage for cost-efficiency while minimizing compute and networking costs:
- Compute: Decoupling compute from storage, Bufstream frees you from having to overprovision either.
- Storage: Bufstream brokers use object storage for Kafka data, saving cost over expensive EBS volumes.
- Networking: Using object storage for Kafka data and zone-aware routing for Kafka clients, Bufstream only incurs inter-zone networking fees for minimal communications between etcd nodes.
- Buf service fee: In production, Buf charges a fee of $0.002 per GiB of uncompressed writes.
Cost breakdown#
| Item | Cost |
|---|---|
| Compute | 1,112 |
| 6 m6in.xlarge Bufstream instances1 | 890 |
| 3 m6in.large etcd instances | 222 |
| Storage | 4,625 |
| S3 operations (approx. 200/second - 125 reads, 75 writes) | 1,127 |
| S3 storage (1532 TiB actual usage) | 3,498 |
| Networking | 226 |
| Inter-zone metadata traffic: ~4.5 MiB/s @ $0.02/GiB ($0.01 egress + $0.01 ingress) | 226 |
| Buf service fees | 5,184 |
| $0.002 per GiB uncompressed write applied to benchmark workload volume | 5,184 |
| Total: | $11,147 |
8x cheaper than Apache Kafka#
Apache Kafka costs $84,796/month for the same workload—7.6x the cost of Bufstream.
Kafka cost problems#
Apache Kafka's architecture couples compute and storage costs, while producers and replication incur inter-zone networking fees that don't apply to Bufstream:
- Compute: For workloads like this, storage drives Kafka's compute footprint: adding storage capacity requires adding additional compute capacity, even if it's unnecessary.
- Storage: Kafka storage requires you to add disks to every broker in the cluster, paying for replication instead of relying on cost-effective object storage.
- Networking: Cost-conscious engineers use Kafka's fetch-from-follower functionality, but producers must always write to the partition leader: 66% of the time, this incurs inter-zone networking fees. Worse, the leader must replicate to brokers in two other zones!
Cost breakdown#
| Item | Cost |
|---|---|
| Compute | 8,039 |
|
57 r4.xlarge1
|
8,039 |
| Storage | 42,025 |
906 TiB provisioned to meet workload and AWS recommendations3
|
42,025 |
| Networking | 34,732 |
|
Monthly volume: 1.66 PiB at $0.02/GiB
|
34,732 |
| Total: | $84,796 |
Conclusion#
Bufstream enables dramatic cost optimization without performance or compatibility risks.
For organizations running high-throughput streaming workloads, Bufstream delivers enterprise-grade capabilities at a fraction of traditional costs with a minimal latency trade-off appropriate for most analytic use cases:
- Full Kafka protocol compatibility eliminates migration risks and enables immediate adoption.
- 8x cost reduction provides immediate savings.
- Cloud-native design reduces operational complexity compared to managing Kafka clusters.
- Additional value for Protobuf users through enhanced data quality and governance features.
-
1-year commitment with no upfront payment using AWS us-east-1 pricing at the time of benchmark. ↩
-
591 TiB uncompressed, compressed to 148 TiB (4:1 ratio) plus 5 TiB metadata and expired records not yet purged. ↩
-
At 1 GiB/s of writes with a 7-day retention period, our data set is 605 TiB uncompressed/151 TiB compressed at 4:1. With three replicas and the AWS-recommended 50% utilization target, this equals 906 TiB of EBS storage. ↩
-
AWS Managed Streaming for Apache Kafka (MSK) caps EBS volumes at 16 TiB. ↩