feat(cli): add volcengine storage service tos:// support

[chenat9]

Summary

This PR introduces support for ByteDance Volcengine TOS object storage as a remote backend for the MCAP CLI, specifically enabling tos:// URI reads.

Motivation

At ByteDance, we extensively use MCAP files stored on TOS for our robotics and VLA (Vision-Language-Action) models. Currently, our internal workflows and tools often require downloading full MCAP files locally before processing. This approach incurs significant bandwidth costs and latency.

To optimize this and avoid maintaining a hard fork of the CLI, we aim to upstream our internal integration to the community. This allows users to inspect and process MCAP files directly from TOS without intermediate downloading.

Implementation Details

• Added a tos:// scheme handler to the CLI's remote file reader.
• The implementation limits dependencies and strictly follows the existing patterns established by the S3 and GCS integrations to ensure consistency.

Future Roadmap

This PR serves as the initial minimal contribution to establish the workflow. We have developed several advanced optimizations internally for AI training scenarios, which we plan to contribute in subsequent PRs:

• High-Performance Access: Integration with TOS accelerator (SSD layer) to provide higher IOPS for training clusters.
• Smart Caching: Optimizing range requests to accelerate random reads (critical for AI data shuffling).
• Write Support: Implementing append-only write support for handling file merging directly on object storage.
• Network Optimization: High-speed network access (similar to AWS CRT) .
• HDFS Support: Remote read implementation for HDFS protocols.

Verification

• Verified locally by building the CLI and running mcap info tos://bucket/path/to/file.mcap.
• Ensured go.mod dependencies are minimal.

First time contributor here, happy to join the community! Looking forward to your review.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[cursor]

Empty region option unconditionally passed to SDK

Medium Severity

The error message states "TOS endpoint or region must be configured", indicating that providing only an endpoint should be valid. However, tos.WithRegion(region) is unconditionally added to the options slice, even when region is an empty string. When only endpoint is configured (and region is empty), this passes WithRegion("") to the SDK, which may cause the client to fail with an invalid region error or exhibit unexpected behavior. The region option should only be added when region != "".

Additional Locations (1)

• go/cli/mcap/utils/readers/tos.go#L28-L33

 

[jneless]

Great Cunge, Thanks for contributing.

I would like to provide more context for mcap community, regarding tos & mcap senarios.
see: https://docs.google.com/document/d/1DFqkflCGO6gqZKQM1DprMzQv30HeuoNRclvG9orWaYs/edit?usp=sharing

maybe the community is hard to review and test in person.

Any review need more test, do not hesitate to email jialin.li@bytedance.com

I would be very happy to provide a pair of AK/SK and demo Endpoint files.