bufstream migrate

Migrate Bufstream cluster storage

Usage

$ bufstream migrate [flags]

Description

Performs migrations on the underlying Bufstream metadata and data storage implementations

Flags

-h, --help

help for migrate

Flags inherited from parent commands

--available-memory uint

Maximum memory available to the process

-c, --config string

Path to config file

--config.actors strings

The actors that are enabled on this agent.

--config.archive.complete_delay_max duration

The maximum time before an archive upload is completed.

--config.archive.concurrent_max int32

The maximum number of topic/partitions to archive at once. When unset (or 0), the default limit is used. When -1, no limit is enforced. When >0, only that many topic/partitions are archived at once per node.

--config.archive.fetch_sync

Whether archive fetches should be synchronized to read at the same log append time.

--config.archive.follow_active

Whether archive creation should try to read/write from the last active zone.

--config.archive.idle_max duration

The duration to wait for more data before completing an archive.

--config.archive.max_bytes int

The maximum size of an archive.

--config.archive.min_bytes int

Determines when to start writing an archive for any topart.

--config.archive.start_delay_max duration

How often to check a topic/partition to start a new archive.

--config.auto_migrate_metadata_storage

If true, the agent will run migrations for the metadata storage on startup.

--config.available_memory_bytes uint

The maximum amount of memory bufstream should consider usable.

--config.cache.archive_max_bytes int

The maximum number of archive log entry bytes to keep in memory.

--config.cache.fetch_record_max_bytes int

The maximum number of record bytes fetched from recent or shelf messages to keep in memory.

--config.cache.intake_max_bytes int

The maximum number of intake file bytes to keep in memory.

--config.cache.kafka_fetch_eager_max_bytes int

The maximum number of record bytes to keep in memory for eagerly fetched records.

--config.cache.producer_max int32

The maximum number of producers tracked per topic/partition. (May be exceeded due to other constraints.)

--config.cache.shelf_max_bytes int

The maximum number of shelf bytes to keep in memory.

--config.cluster string

The name of the cluster.

--config.connect_address.host string

A hostname or IP address to connect to.

--config.connect_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.connect_http_version string

The HTTP version to use for inter-agent Connect RPCs.

--config.connect_isolation

Whether inter-agent Connect clients should be unique for reads and writes.

--config.connect_public_address.host string

A hostname or IP address to connect to.

--config.connect_public_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.debug.checkpoint_error.read_error_probability float32

--config.debug.checkpoint_error.write_error_probability float32

--config.debug.cluster_error.read_error_probability float32

--config.debug.cluster_error.write_error_probability float32

--config.debug.datastore_error.read_error_probability float32

--config.debug.datastore_error.write_error_probability float32

--config.debug.log_api_requests_regex string

--config.debug.log_api_responses_regex string

--config.disabled_actors strings

The actors that are disabled on this agent.

--config.dispatch.local_fetch

Whether calls to fetch records should be handled separately by each node or sharded among multiple nodes.

--config.dispatch.local_intake_cache

Whether the intake cache should be handled separately by each node or sharded among multiple nodes.

--config.dispatch.local_produce

Whether calls to produce records should be handled separately by each node or sharded among multiple nodes.

--config.dispatch.unavailable_retry_count int32

The number of retry attempts to make when an Unavailable error is encountered.

--config.etcd.session_ttl_seconds int32

The amount of time an etcd node can be unreachable before it is considered down.

--config.fetch_sync_group_count int32

The number of 'groups' to cluster fetchers into for synchronization at the same log append time.

--config.in_memory

If true, the agent will use an in-memory cache for metadata storage.

--config.intake.delay_max duration

The maximum delay to wait before writing an intake file.

--config.intake.delay_max_bytes int

The maximum number of bytes to enqueue before writing an intake file.

--config.intake.file_delete_delay_max duration

How often to scan all intake files to (try to) delete old files.

--config.intake.log_append_time_difference_max duration

The maximum difference between intake write time and log append time.

--config.intake.producer_id_batch_size int32

How many producer IDs a Bufstream process reserves at a time.

--config.intake.recent_msg_min int32

The minimum number of recent messages to keep for each topic/partition.

--config.intake.recent_sequence_eager

Whether recent messages should be sequenced actively.

--config.intake.shelf_msg_max int32

The maximum number of recent messages to shelve in at a time.

--config.intake.txn_timeout_default duration

The default timeout for a new transactions.

--config.intake.txn_timeout_max duration

The maximum timeout for all transactions.

--config.intake.write_stream

Whether intake entries should be streamed when written.

--config.intake.write_stream_chunk_bytes int32

The maximum number of bytes to write in a single intake write stream chunk.

--config.intake.write_through_cache

Whether intake entries should be written through the cache.

--config.kafka.address.host string

A hostname or IP address to connect to.

--config.kafka.address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.kafka.exact_log_sizes

If exact log sizes should be fetched when listing sizes for all topics/partitions.

--config.kafka.fetch_eager

If a fetch should return as soon as any records are available.

--config.kafka.fetch_sync

If fetches from different readers should be synchronized to improve cache hit rates.

--config.kafka.group_consumer_session_timeout duration

The default group consumer session timeout.

--config.kafka.group_consumer_session_timeout_max duration

The maximum group consumer session timeout.

--config.kafka.group_consumer_session_timeout_min duration

The minimum group consumer session timeout.

--config.kafka.idle_timeout duration

How long a Kafka connection can be idle before being closed by the server.

--config.kafka.num_partitions int32

The default number of partitions to use for a new topic.

--config.kafka.partition_balance_strategy string

How to balance topic/partitions across bufstream nodes.

--config.kafka.produce_concurrent

If records from a producer to different topic/partitions may be sequenced concurrently instead of serially.

--config.kafka.public_address.host string

A hostname or IP address to connect to.

--config.kafka.public_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.kafka.request_buffer_size uint32

The number of kafka request to unmarshal and buffer before processing.

--config.kafka.tls.client_auth string

Declare the policy the server will follow for mutual TLS (mTLS).

--config.kafka.zone_balance_strategy string

How to balancing clients across zones, when client does not specify a zone.

--config.labels stringToString

Labels associated with the Bufstream agent.

--config.name string

The name of this Bufstream agent.

--config.observability.debug_address.host string

A hostname or IP address to connect to.

--config.observability.debug_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.observability.log_format string

log format, defaults to TEXT when connected to a terminal, otherwise JSON.

--config.observability.log_git_version

If set, include "version=" in log output.

--config.observability.log_level string

log level, defaults to INFO

--config.observability.metrics_exporter string

OpenTelemetry exporter for metrics, defaults to NONE.

--config.observability.trace_exporter string

OpenTelemetry exporter for traces, defaults to NONE.

--config.observability.trace_ratio float

OpenTelemetry trace sample ratio, defaults to 1.

--config.postgres.alloy_db_proxy_icn.bytes bytesBase64

An inline byte blob of the data

--config.postgres.alloy_db_proxy_icn.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.postgres.alloy_db_proxy_icn.env_var string

An environment variable containing the data.

--config.postgres.alloy_db_proxy_icn.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.postgres.alloy_db_proxy_icn.string string

An inline string of the data

--config.postgres.cloud_sql_proxy_icn.bytes bytesBase64

An inline byte blob of the data

--config.postgres.cloud_sql_proxy_icn.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.postgres.cloud_sql_proxy_icn.env_var string

An environment variable containing the data.

--config.postgres.cloud_sql_proxy_icn.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.postgres.cloud_sql_proxy_icn.string string

An inline string of the data

--config.postgres.dsn.bytes bytesBase64

An inline byte blob of the data

--config.postgres.dsn.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.postgres.dsn.env_var string

An environment variable containing the data.

--config.postgres.dsn.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.postgres.dsn.string string

An inline string of the data

--config.record_expiry_delay_max duration

How often to scan all owned partitions to (try to) delete expired records.

--config.storage.access_key_id.bytes bytesBase64

An inline byte blob of the data

--config.storage.access_key_id.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.storage.access_key_id.env_var string

An environment variable containing the data.

--config.storage.access_key_id.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.storage.access_key_id.string string

An inline string of the data

--config.storage.bucket string

The object storage bucket where data is stored.

--config.storage.debug_logging string

Enables data storage debug logging at the specified level.

--config.storage.endpoint string

The provider's HTTPS endpoint to use instead of the default

--config.storage.force_path_style

Enable path-based routing (instead of subdomains) for buckets.

--config.storage.hedge_timeout duration

How long before a GET request to the data storage provider is hedged with an additional request.

--config.storage.prefix string

The path prefix of objects stored in the data storage.

--config.storage.provider string

The data storage provider.

--config.storage.put_hedge_delay_ms int

Enables hedging of PUT requests to the data storage provider with the specified delay.

--config.storage.region bucket

The region in which the bucket exists.

--config.storage.secret_access_key.bytes bytesBase64

An inline byte blob of the data

--config.storage.secret_access_key.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.storage.secret_access_key.env_var string

An environment variable containing the data.

--config.storage.secret_access_key.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.storage.secret_access_key.string string

An inline string of the data

--config.storage.write_isolation

If writes should use the same clients as reads.

--config.zone string

The location of the agent, e.g., the datacenter/availability zone where the agent is running.

--etcd.address strings

Address(es) of an etcd cluster

--inmemory

Use in-memory storage

--nokafka

Start without Kafka interface

Parent Command

• bufstream - The Bufstream process.