Skip to content

Conversation

@haacked
Copy link
Contributor

@haacked haacked commented Oct 31, 2025

Problem

The Rust RedisClient needs to be compatible with Django's Redis caching behavior for seamless interoperability between Python and Rust services. Currently:

  1. No compression support - Django uses Zstd compression for large values (>512 bytes) to reduce Redis memory usage and network overhead
  2. Limited serialization options - Only Pickle format is supported, but different use cases need different formats (UTF-8 for simple strings, raw bytes for binary data)
  3. Inconsistent configuration - No centralized way to configure compression and serialization behavior per client

Without these features, Rust services cannot properly read/write data that Django services cache, and vice versa.

Changes

Added configurable compression and serialization format support to RedisClient with Django-compatible defaults:

Compression Support (Zstd)

  • CompressionConfig - Configures compression behavior with Django-compatible defaults:
    • enabled: true - Compression on by default
    • threshold: 512 - Matches Django's ZstdCompressor.min_length
    • level: 0 - Matches Django's default preset (equivalent to level 3)
  • Graceful decompression - try_decompress() handles both compressed and uncompressed data, allowing gradual rollout
  • Automatic compression - Values over threshold are automatically compressed on write

Serialization Format Support

  • RedisValueFormat enum - Three formats for different use cases:
    • Pickle (default) - Django-compatible serialization
    • Utf8 - Simple string encoding
    • RawBytes - Direct binary data (via dedicated methods)
  • Per-client configuration - Format is set once at client creation
  • Per-call overrides - *_with_format() methods available when needed

API Design

Simplified constructor API:

// Default: compression disabled, Pickle format
let client = RedisClient::new(addr).await?;

// Full control over both settings
let client = RedisClient::with_config(
    addr,
    CompressionConfig::new(true, 512, 0),
    RedisValueFormat::Utf8,
).await?;

// Django-compatible: compression enabled, Pickle serialization
let client = RedisClient::with_config(
    addr,
    CompressionConfig::default(),  // 512-byte threshold, level 0
    RedisValueFormat::Pickle,
).await?;

How did you test this code?

cargo test --package common-redis --lib  # All 20 tests pass
cargo clippy --package common-redis --all-targets --all-features -- -D warnings  # No warnings

Tests cover:

  • Compression behavior at various thresholds (below, at, above)
  • Compression levels (0, custom levels)
  • Round-trip serialization for all formats (Pickle, Utf8)
  • Format interoperability (ensuring different formats produce different outputs)
  • Decompression of both compressed and uncompressed data
  • Error handling for invalid formats

Backwards Compatible

Existing code is not affected as defaults remain the same. Added a new ctor for configuring the client.

@haacked haacked requested a review from Copilot October 31, 2025 22:10
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds zstd compression support to the Redis client library to match Django's compression behavior. The implementation allows for transparent compression/decompression of Redis values based on configurable thresholds.

  • Adds CompressionConfig struct with Django-compatible defaults (512-byte threshold, level 0)
  • Implements compression/decompression logic that gracefully handles both compressed and uncompressed data
  • Updates RedisClient to store and use compression and format configurations

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.

File Description
rust/common/redis/src/lib.rs Adds compression infrastructure, configuration types, helper methods, and comprehensive test coverage
rust/common/redis/Cargo.toml Adds zstd dependency version 0.13
rust/Cargo.lock Updates lock file with zstd dependency

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@haacked haacked force-pushed the haacked/redis-client-ztsd branch 3 times, most recently from eae6603 to 10e2bb9 Compare October 31, 2025 22:27
@haacked haacked force-pushed the haacked/redis-client-ztsd branch from 10e2bb9 to 535386f Compare October 31, 2025 22:47
@haacked haacked marked this pull request as ready for review October 31, 2025 22:59
@haacked haacked requested review from a team and oliverb123 October 31, 2025 22:59
@posthog-project-board-bot posthog-project-board-bot bot moved this to In Review in Feature Flags Oct 31, 2025
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

I don't want to break other consumers of this library.
Addresses feedback from PR review:
- Always call `try_decompress()` on reads, regardless of compression config
- This allows clients to read data written with different compression settings
- Prevents "config mismatch" issues where data becomes unreadable

Also improves error detection:
- Add logging via tracing to detect corrupted compressed data
- Check for zstd magic bytes to distinguish corruption from uncompressed data
- Log warning only when decompression fails on data with compression headers
Copy link
Contributor

@gustavohstrassburger gustavohstrassburger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, I just noticed that the description of the PR mentions that CompressionConfig is enabled by default and you've changed that, so you might want to change there also.

@github-project-automation github-project-automation bot moved this from In Review to Approved in Feature Flags Nov 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants