bufstream archive
Archive a topic/partition
Usage
$ bufstream archive [flags]
Description
Cause the Bufstream cluster to archive the given topic/partition
Flags
--block
Block until the archive is complete
-h, --help
help for archive
--partition int32
The partition to archive
--topic string
The topic to archive
Flags inherited from parent commands
--available-memory uint
Maximum memory available to the process
-c, --config string
Path to config file
--config.actors strings
The actors that are enabled on this agent.
--config.archive.complete_delay_max duration
The maximum time before an archive upload is completed.
--config.archive.concurrent_max int32
The maximum number of topic/partitions to archive at once. When unset (or 0), the default limit is used. When -1, no limit is enforced. When >0, only that many topic/partitions are archived at once per node.
--config.archive.fetch_sync
Whether archive fetches should be synchronized to read at the same log append time.
--config.archive.follow_active
Whether archive creation should try to read/write from the last active zone.
--config.archive.idle_max duration
The duration to wait for more data before completing an archive.
--config.archive.max_bytes int
The maximum size of an archive.
--config.archive.min_bytes int
Determines when to start writing an archive for any topart.
--config.archive.start_delay_max duration
How often to check a topic/partition to start a new archive.
--config.auto_migrate_metadata_storage
If true, the agent will run migrations for the metadata storage on startup.
--config.available_memory_bytes uint
The maximum amount of memory bufstream should consider usable.
--config.cache.archive_max_bytes int
The maximum number of archive log entry bytes to keep in memory.
--config.cache.fetch_record_max_bytes int
The maximum number of record bytes fetched from recent or shelf messages to keep in memory.
--config.cache.intake_max_bytes int
The maximum number of intake file bytes to keep in memory.
--config.cache.kafka_fetch_eager_max_bytes int
The maximum number of record bytes to keep in memory for eagerly fetched records.
--config.cache.producer_max int32
The maximum number of producers tracked per topic/partition. (May be exceeded due to other constraints.)
--config.cache.shelf_max_bytes int
The maximum number of shelf bytes to keep in memory.
--config.cluster string
The name of the cluster.
--config.connect_address.host string
A hostname or IP address to connect to.
--config.connect_address.port uint32
The associated port. If unspecified, refer to the field documentation for default behavior.
--config.connect_http_version string
The HTTP version to use for inter-agent Connect RPCs.
--config.connect_isolation
Whether inter-agent Connect clients should be unique for reads and writes.
--config.connect_public_address.host string
A hostname or IP address to connect to.
--config.connect_public_address.port uint32
The associated port. If unspecified, refer to the field documentation for default behavior.
--config.debug.checkpoint_error.read_error_probability float32
--config.debug.checkpoint_error.write_error_probability float32
--config.debug.cluster_error.read_error_probability float32
--config.debug.cluster_error.write_error_probability float32
--config.debug.datastore_error.read_error_probability float32
--config.debug.datastore_error.write_error_probability float32
--config.debug.log_api_requests_regex string
--config.debug.log_api_responses_regex string
--config.disabled_actors strings
The actors that are disabled on this agent.
--config.dispatch.local_fetch
Whether calls to fetch records should be handled separately by each node or sharded among multiple nodes.
--config.dispatch.local_intake_cache
Whether the intake cache should be handled separately by each node or sharded among multiple nodes.
--config.dispatch.local_produce
Whether calls to produce records should be handled separately by each node or sharded among multiple nodes.
--config.dispatch.unavailable_retry_count int32
The number of retry attempts to make when an Unavailable error is encountered.
--config.etcd.session_ttl_seconds int32
The amount of time an etcd node can be unreachable before it is considered down.
--config.fetch_sync_group_count int32
The number of 'groups' to cluster fetchers into for synchronization at the same log append time.
--config.in_memory
If true, the agent will use an in-memory cache for metadata storage.
--config.intake.delay_max duration
The maximum delay to wait before writing an intake file.
--config.intake.delay_max_bytes int
The maximum number of bytes to enqueue before writing an intake file.
--config.intake.file_delete_delay_max duration
How often to scan all intake files to (try to) delete old files.
--config.intake.log_append_time_difference_max duration
The maximum difference between intake write time and log append time.
--config.intake.producer_id_batch_size int32
How many producer IDs a Bufstream process reserves at a time.
--config.intake.recent_msg_min int32
The minimum number of recent messages to keep for each topic/partition.
--config.intake.recent_sequence_eager
Whether recent messages should be sequenced actively.
--config.intake.shelf_msg_max int32
The maximum number of recent messages to shelve in at a time.
--config.intake.txn_timeout_default duration
The default timeout for a new transactions.
--config.intake.txn_timeout_max duration
The maximum timeout for all transactions.
--config.intake.write_stream
Whether intake entries should be streamed when written.
--config.intake.write_stream_chunk_bytes int32
The maximum number of bytes to write in a single intake write stream chunk.
--config.intake.write_through_cache
Whether intake entries should be written through the cache.
--config.kafka.address.host string
A hostname or IP address to connect to.
--config.kafka.address.port uint32
The associated port. If unspecified, refer to the field documentation for default behavior.
--config.kafka.exact_log_sizes
If exact log sizes should be fetched when listing sizes for all topics/partitions.
--config.kafka.fetch_eager
If a fetch should return as soon as any records are available.
--config.kafka.fetch_sync
If fetches from different readers should be synchronized to improve cache hit rates.
--config.kafka.group_consumer_session_timeout duration
The default group consumer session timeout.
--config.kafka.group_consumer_session_timeout_max duration
The maximum group consumer session timeout.
--config.kafka.group_consumer_session_timeout_min duration
The minimum group consumer session timeout.
--config.kafka.idle_timeout duration
How long a Kafka connection can be idle before being closed by the server.
--config.kafka.num_partitions int32
The default number of partitions to use for a new topic.
--config.kafka.partition_balance_strategy string
How to balance topic/partitions across bufstream nodes.
--config.kafka.produce_concurrent
If records from a producer to different topic/partitions may be sequenced concurrently instead of serially.
--config.kafka.public_address.host string
A hostname or IP address to connect to.
--config.kafka.public_address.port uint32
The associated port. If unspecified, refer to the field documentation for default behavior.
--config.kafka.request_buffer_size uint32
The number of kafka request to unmarshal and buffer before processing.
--config.kafka.tls.client_auth string
Declare the policy the server will follow for mutual TLS (mTLS).
--config.kafka.zone_balance_strategy string
How to balancing clients across zones, when client does not specify a zone.
--config.labels stringToString
Labels associated with the Bufstream agent.
--config.name string
The name of this Bufstream agent.
--config.observability.debug_address.host string
A hostname or IP address to connect to.
--config.observability.debug_address.port uint32
The associated port. If unspecified, refer to the field documentation for default behavior.
--config.observability.log_format string
log format, defaults to TEXT when connected to a terminal, otherwise JSON.
--config.observability.log_git_version
If set, include "version=
--config.observability.log_level string
log level, defaults to INFO
--config.observability.metrics_exporter string
OpenTelemetry exporter for metrics, defaults to NONE.
--config.observability.trace_exporter string
OpenTelemetry exporter for traces, defaults to NONE.
--config.observability.trace_ratio float
OpenTelemetry trace sample ratio, defaults to 1.
--config.postgres.alloy_db_proxy_icn.bytes bytesBase64
An inline byte blob of the data
--config.postgres.alloy_db_proxy_icn.encoding string
The encoding of the data source value. Defaults to PLAINTEXT.
--config.postgres.alloy_db_proxy_icn.env_var string
An environment variable containing the data.
--config.postgres.alloy_db_proxy_icn.path string
A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.
--config.postgres.alloy_db_proxy_icn.string string
An inline string of the data
--config.postgres.cloud_sql_proxy_icn.bytes bytesBase64
An inline byte blob of the data
--config.postgres.cloud_sql_proxy_icn.encoding string
The encoding of the data source value. Defaults to PLAINTEXT.
--config.postgres.cloud_sql_proxy_icn.env_var string
An environment variable containing the data.
--config.postgres.cloud_sql_proxy_icn.path string
A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.
--config.postgres.cloud_sql_proxy_icn.string string
An inline string of the data
--config.postgres.dsn.bytes bytesBase64
An inline byte blob of the data
--config.postgres.dsn.encoding string
The encoding of the data source value. Defaults to PLAINTEXT.
--config.postgres.dsn.env_var string
An environment variable containing the data.
--config.postgres.dsn.path string
A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.
--config.postgres.dsn.string string
An inline string of the data
--config.record_expiry_delay_max duration
How often to scan all owned partitions to (try to) delete expired records.
--config.storage.access_key_id.bytes bytesBase64
An inline byte blob of the data
--config.storage.access_key_id.encoding string
The encoding of the data source value. Defaults to PLAINTEXT.
--config.storage.access_key_id.env_var string
An environment variable containing the data.
--config.storage.access_key_id.path string
A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.
--config.storage.access_key_id.string string
An inline string of the data
--config.storage.bucket string
The object storage bucket where data is stored.
--config.storage.debug_logging string
Enables data storage debug logging at the specified level.
--config.storage.endpoint string
The provider's HTTPS endpoint to use instead of the default
--config.storage.force_path_style
Enable path-based routing (instead of subdomains) for buckets.
--config.storage.hedge_timeout duration
How long before a GET request to the data storage provider is hedged with an additional request.
--config.storage.prefix string
The path prefix of objects stored in the data storage.
--config.storage.provider string
The data storage provider.
--config.storage.put_hedge_delay_ms int
Enables hedging of PUT requests to the data storage provider with the specified delay.
--config.storage.region bucket
The region in which the bucket exists.
--config.storage.secret_access_key.bytes bytesBase64
An inline byte blob of the data
--config.storage.secret_access_key.encoding string
The encoding of the data source value. Defaults to PLAINTEXT.
--config.storage.secret_access_key.env_var string
An environment variable containing the data.
--config.storage.secret_access_key.path string
A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.
--config.storage.secret_access_key.string string
An inline string of the data
--config.storage.write_isolation
If writes should use the same clients as reads.
--config.zone string
The location of the agent, e.g., the datacenter/availability zone where the agent is running.
--etcd.address strings
Address(es) of an etcd cluster
--inmemory
Use in-memory storage
--nokafka
Start without Kafka interface
Parent Command
- bufstream - The Bufstream process.