bufstream migrate

Migrate Bufstream cluster storage

Usage

$ bufstream migrate [flags]

Description

Performs migrations on the underlying Bufstream metadata and data storage implementations

Flags

--available-memory uint

Maximum memory available to the process

-c, --config string

Path to config file

--config.actors strings

The actors that are enabled on this agent.

--config.admin_address.host string

A hostname or IP address to connect to or listen on.

--config.admin_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.admin_tls.client_auth string

Declare the policy the server will follow for mutual TLS (mTLS).

--config.archive.complete_delay_max duration

The maximum time before an archive upload is completed.

--config.archive.concurrent_max int32

The maximum number of topic/partitions to archive at once.

--config.archive.default_log_level string

The default log level for background archive operations.

--config.archive.fetch_max_batches int32

The maximum number of batches to fetch from an archive in a single request, per topic/partition.

--config.archive.fetch_sync

Whether archive fetches should be synchronized to read at the same log append time.

--config.archive.follow_active

Whether archive creation should try to read/write from the last active zone.

--config.archive.idle_max duration

The duration to wait for more data before completing an archive.

--config.archive.max_bytes int

The maximum size of an archive.

--config.archive.min_bytes int

Determines when to start writing an archive for any topart.

--config.archive.start_delay_max duration

How often to check a topic/partition to start a new archive.

--config.auto_migrate_metadata_storage

If true, the agent will run migrations for the metadata storage on startup.

--config.available_memory_bytes uint

The maximum amount of memory bufstream should consider usable.

--config.cache.archive_max_bytes int

The maximum number of archive log entry bytes to keep in memory.

--config.cache.fetch_record_max_bytes int

The maximum number of record bytes fetched from recent or shelf messages to keep in memory.

--config.cache.intake_max_bytes int

The maximum number of intake file bytes to keep in memory.

--config.cache.kafka_fetch_eager_max_bytes int

The maximum number of record bytes to keep in memory for eagerly fetched records.

--config.cache.producer_max int32

The maximum number of producers tracked per topic/partition. (May be exceeded due to other constraints.)

--config.cache.shelf_max_bytes int

The maximum number of shelf bytes to keep in memory.

--config.cluster string

The name of the cluster.

--config.connect_address.host string

A hostname or IP address to connect to or listen on.

--config.connect_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.connect_http_version string

The HTTP version to use for inter-agent Connect RPCs.

--config.connect_isolation

Whether inter-agent Connect clients should be unique for reads and writes.

--config.connect_public_address.host string

A hostname or IP address to connect to or listen on.

--config.connect_public_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.connect_tls.client.certificate.chain.bytes bytesBase64

An inline byte blob of the data

--config.connect_tls.client.certificate.chain.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.connect_tls.client.certificate.chain.env_var string

An environment variable containing the data.

--config.connect_tls.client.certificate.chain.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.connect_tls.client.certificate.chain.string string

An inline string of the data

--config.connect_tls.client.certificate.private_key.bytes bytesBase64

An inline byte blob of the data

--config.connect_tls.client.certificate.private_key.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.connect_tls.client.certificate.private_key.env_var string

An environment variable containing the data.

--config.connect_tls.client.certificate.private_key.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.connect_tls.client.certificate.private_key.string string

An inline string of the data

--config.connect_tls.client.insecure_skip_verify

Controls whether a client verifies the server's certificate chain and host name. If true, the dialer accepts any certificate presented by the server and host name in that certificate. In this mode, TLS is susceptible to machine-in-the-middle attacks and should only be used for testing.

--config.connect_tls.server.client_auth string

Declare the policy the server will follow for mutual TLS (mTLS).

--config.connect_tls.server_name string

The server name clients should use for TLS verification when connecting to this ConnectRPC server. If unset, the listener address will be used.

--config.debug.checkpoint_error.read_error_probability float32

--config.debug.checkpoint_error.write_error_probability float32

--config.debug.cluster_error.read_error_probability float32

--config.debug.cluster_error.write_error_probability float32

--config.debug.consumer_offset_stride int32

--config.debug.datastore_error.read_error_probability float32

--config.debug.datastore_error.write_error_probability float32

--config.debug.log_api_requests_regex string

--config.debug.log_api_responses_regex string

--config.disabled_actors strings

The actors that are disabled on this agent.

--config.dispatch.local_fetch_archive

Whether calls to fetch archive records should be handled separately by each agent or dispatched to shared agents.

--config.dispatch.local_fetch_recent

Whether calls to fetch recent records should be handled separately by each agent or dispatched to shared agents.

--config.dispatch.local_intake_cache

Whether the intake cache should be handled separately by each agent or dispatched to shared agents.

--config.dispatch.local_produce

Whether calls to produce records should be handled separately by each agent or dispatched to shared agents.

--config.dispatch.unavailable_retry_count int32

The number of retry attempts to make when an Unavailable error is encountered.

--config.etcd.session_ttl_seconds int32

The amount of time an etcd node can be unreachable before it is considered down.

--config.etcd.tls.certificate.chain.bytes bytesBase64

An inline byte blob of the data

--config.etcd.tls.certificate.chain.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.etcd.tls.certificate.chain.env_var string

An environment variable containing the data.

--config.etcd.tls.certificate.chain.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.etcd.tls.certificate.chain.string string

An inline string of the data

--config.etcd.tls.certificate.private_key.bytes bytesBase64

An inline byte blob of the data

--config.etcd.tls.certificate.private_key.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.etcd.tls.certificate.private_key.env_var string

An environment variable containing the data.

--config.etcd.tls.certificate.private_key.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.etcd.tls.certificate.private_key.string string

An inline string of the data

--config.etcd.tls.insecure_skip_verify

Controls whether a client verifies the server's certificate chain and host name. If true, the dialer accepts any certificate presented by the server and host name in that certificate. In this mode, TLS is susceptible to machine-in-the-middle attacks and should only be used for testing.

--config.fetch_sync_group_count int32

The number of 'groups' to cluster fetchers into for synchronization at the same log append time.

--config.in_memory

If true, the agent will use an in-memory cache for metadata storage.

--config.intake.delay_max duration

The maximum delay to wait before writing an intake file.

--config.intake.delay_max_bytes int

The maximum number of bytes to enqueue before writing an intake file.

--config.intake.end_txn_revision_check

Whether to record the revision that the end transaction request was started at and to fail the request if the transaction changed while active since then.

--config.intake.end_txn_skew_max duration

The maximum amount of time an end transaction request can appear to come before the last modification to the transaction.

--config.intake.file_delete_delay_max duration

How often to scan all intake files to (try to) delete old files.

--config.intake.log_append_time_difference_max duration

The maximum difference between intake write time and log append time.

--config.intake.producer_id_batch_size int32

How many producer IDs a Bufstream process reserves at a time.

--config.intake.recent_msg_min int32

The minimum number of recent messages to keep for each topic/partition.

--config.intake.recent_sequence_eager

Whether recent messages should be sequenced actively.

--config.intake.shelf_msg_max int32

The maximum number of recent messages to shelve in at a time.

--config.intake.txn_timeout_default duration

The default timeout for a new transactions.

--config.intake.txn_timeout_max duration

The maximum timeout for all transactions.

--config.intake.write_stream

Whether intake entries should be streamed when written.

--config.intake.write_stream_chunk_bytes int32

The maximum number of bytes to write in a single intake write stream chunk.

--config.intake.write_through_cache

Whether intake entries should be written through the cache.

--config.kafka.address.host string

A hostname or IP address to connect to or listen on.

--config.kafka.address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.kafka.default_broker_count int32

The default number of brokers to report to clients.

--config.kafka.distinct_hosts

If the casing of hostnames should be randomized per 'broker'.

--config.kafka.exact_log_offsets

If exact log high water mark and start offsets should be computed when fetching records.

--config.kafka.exact_log_sizes

If exact log sizes should be fetched when listing sizes for all topics/partitions.

--config.kafka.fetch_eager

If a fetch should return as soon as any records are available.

--config.kafka.fetch_eager_offset_strategy string

The strategy to use when no data is available for a topic partition.

--config.kafka.fetch_sync

If fetches from different readers should be synchronized to improve cache hit rates.

--config.kafka.group_consumer_session_timeout duration

The default group consumer session timeout.

--config.kafka.group_consumer_session_timeout_max duration

The maximum group consumer session timeout.

--config.kafka.group_consumer_session_timeout_min duration

The minimum group consumer session timeout.

--config.kafka.idle_timeout duration

How long a Kafka connection can be idle before being closed by the server.

--config.kafka.num_partitions int32

The default number of partitions to use for a new topic.

--config.kafka.partition_balance_strategy string

How to balance topic/partitions across bufstream agents.

--config.kafka.produce_concurrent

If records from a producer to different topic/partitions may be sequenced concurrently instead of serially.

--config.kafka.producer_request_timeout duration

The maximum period to wait for a producer RPC to complete.

--config.kafka.public_address.host string

A hostname or IP address to connect to or listen on.

--config.kafka.public_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.kafka.request_buffer_size uint32

The number of Kafka requests to unmarshal and buffer before processing.

--config.kafka.shutdown_grace_period duration

The grace period to allow clients before shutting down.

--config.kafka.tls.client_auth string

Declare the policy the server will follow for mutual TLS (mTLS).

--config.kafka.wait_for_latest

If 'broker' should ensure a topic/partition is fully loaded before serving.

--config.kafka.zone_balance_strategy string

How to balance clients across zones, when then client does not specify a zone.

--config.labels stringToString

Labels associated with the Bufstream agent.

--config.name string

The name of this Bufstream agent.

--config.observability.debug_address.host string

A hostname or IP address to connect to or listen on.

--config.observability.debug_address.port uint32

The associated port. If unspecified, refer to the field documentation for default behavior.

--config.observability.exporter.address string

The default base address used by OTLP_HTTP and OTLP_GRPC exporters, with a host name and an optional port number. For OTLP_HTTP, "/v1/{metrics, traces}" will be appended to this address, unless the path is overriden by metrics.path or traces.path. If port is unspecified, it defaults to 4317 for OTLP_GRPC and 4318 for OTLP_HTTP.

--config.observability.exporter.insecure

If set to true, TLS is disabled for the OTLP exporter. This can be overwritten by metrics.insecure or traces.insecure.

--config.observability.log_format string

log format, defaults to TEXT when connected to a terminal, otherwise JSON.

--config.observability.log_git_version

If set, include "version=" in log output.

--config.observability.log_level string

log level, defaults to INFO

--config.observability.metrics.address string

The endpoint for OTLP exporter, with a host name and an optional port number. If this is not set, it falls back to observability.exporter.address. If that is not set, it falls back to OTEL's default behavior, using the the host and port of OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, if not found then OTEL_EXPORTER_OTLP_ENDPOINT and finally localhost:4318 for OTLP_HTTP or locahost:4317 for OTLP_GRPC.

--config.observability.metrics.exporter_type string

The type of exporter to use.

--config.observability.metrics.insecure

If set to true, TLS is disabled for the OTLP exporter.

--config.observability.metrics.omit_partition_attribute

This omits metrics that depend on the kafka.topic.partition attribute, which may have high cardinality depending on the configuration. One example is kafka.topic.partition.offset.high_water_mark. This omits only the attribute for metrics that have this attribute without depending on it. One example is kafka.produce.record.size.

--config.observability.metrics.path string

This url path used by the OTLP_HTTP exporter, this defaults to "/v1/metrics". This is appended to the host and port of the endpoint that the exporter connects to.

--config.observability.metrics_exporter string

OpenTelemetry exporter for metrics, defaults to NONE. Deprecated: use metrics.exporter_type instead.

--config.observability.sensitive_information_redaction string

Redact sensitive information such as topic names, before adding to to metrics, traces and logs.

--config.observability.trace_exporter string

OpenTelemetry exporter for traces, defaults to NONE. Deprecated: use traces.exporter_type instead.

--config.observability.trace_ratio float

OpenTelemetry trace sample ratio, defaults to 1. Deprecated: use traces.trace_ratio instead.

--config.observability.traces.address string

The endpoint for OTLP exporter, with a host name and an optional port number. If this is not set, it falls back to observability.exporter.address. If that is not set, it falls back to the OTEL's default behavior, using the host and port of OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, if not found then OTEL_EXPORTER_OTLP_ENDPOINT and finally localhost:4318 for OTLP_HTTP or locahost:4317 for OTLP_GRPC.

--config.observability.traces.exporter_type string

The type of exporter to use.

--config.observability.traces.insecure

If set to true, TLS is disabled for the OTLP exporter.

--config.observability.traces.path string

This url path used by the OTLP_HTTP exporter, this defaults to "/v1/traces". This is appended to the host and port of the endpoint that the exporter connects to.

--config.observability.traces.trace_ratio float

OpenTelemetry trace sample ratio, defaults to 1.

--config.postgres.alloy_db_proxy_icn.bytes bytesBase64

An inline byte blob of the data

--config.postgres.alloy_db_proxy_icn.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.postgres.alloy_db_proxy_icn.env_var string

An environment variable containing the data.

--config.postgres.alloy_db_proxy_icn.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.postgres.alloy_db_proxy_icn.string string

An inline string of the data

--config.postgres.cloud_sql_proxy_icn.bytes bytesBase64

An inline byte blob of the data

--config.postgres.cloud_sql_proxy_icn.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.postgres.cloud_sql_proxy_icn.env_var string

An environment variable containing the data.

--config.postgres.cloud_sql_proxy_icn.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.postgres.cloud_sql_proxy_icn.string string

An inline string of the data

--config.postgres.dsn.bytes bytesBase64

An inline byte blob of the data

--config.postgres.dsn.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.postgres.dsn.env_var string

An environment variable containing the data.

--config.postgres.dsn.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.postgres.dsn.string string

An inline string of the data

--config.record_expiry_delay_max duration

How often to scan all owned partitions to (try to) delete expired records.

--config.storage.access_key_id.bytes bytesBase64

An inline byte blob of the data

--config.storage.access_key_id.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.storage.access_key_id.env_var string

An environment variable containing the data.

--config.storage.access_key_id.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.storage.access_key_id.string string

An inline string of the data

--config.storage.bucket string

The object storage bucket where data is stored.

--config.storage.debug_logging string

Enables data storage debug logging at the specified level.

--config.storage.endpoint string

The provider's HTTPS endpoint to use instead of the default

--config.storage.force_path_style

Enable path-based routing (instead of subdomains) for buckets.

--config.storage.get_hedge_delay duration

How long before a GET request to the data storage provider is hedged with an additional request.

--config.storage.prefix string

The path prefix of objects stored in the data storage.

--config.storage.provider string

The data storage provider.

--config.storage.put_hedge_delay duration

Enables hedging of PUT requests to the data storage provider with the specified delay.

--config.storage.region bucket

The region in which the bucket exists.

--config.storage.secret_access_key.bytes bytesBase64

An inline byte blob of the data

--config.storage.secret_access_key.encoding string

The encoding of the data source value. Defaults to PLAINTEXT.

--config.storage.secret_access_key.env_var string

An environment variable containing the data.

--config.storage.secret_access_key.path string

A file path to the data relative to the current working directory. Trailing newlines are stripped from the file contents.

--config.storage.secret_access_key.string string

An inline string of the data

--config.storage.write_isolation

If writes should use the same clients as reads.

--config.zone string

The location of the agent, e.g., the datacenter/availability zone where the agent is running.

--etcd.address strings

Address(es) of an etcd cluster

-h, --help

help for migrate

--inmemory

Use in-memory storage

--nokafka

Start without Kafka interface

Parent Command

• bufstream - The Bufstream process.