VFS Tests: Working Solution with Shared Directory Approach

[corylanou]

✅ Working Solution - Tests Pass!

Ben's clarification was the key: "If I can get the env vars set once then that's good enough. I don't mind using a single temp directory to run tests against."

This completely simplified the problem! No complex lazy initialization or extension loading needed.

The Solution

Use TestMain to set up VFS once with a shared directory:

const sharedReplicaDir = "/tmp/litestream-vfs-test-shared"

func TestMain(m *testing.M) {
    // Create shared replica directory
    os.RemoveAll(sharedReplicaDir)
    os.MkdirAll(sharedReplicaDir, 0755)

    // Set up VFS once for all tests
    client := file.NewReplicaClient(sharedReplicaDir)
    logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{
        Level: slog.LevelDebug,
    }))
    vfs := litestream.NewVFS(client, logger)
    vfs.PollInterval = 100 * time.Millisecond

    // Register VFS once at package level
    sqlite3vfs.RegisterVFS("litestream", vfs)

    // Run all tests
    code := m.Run()

    // Cleanup
    os.RemoveAll(sharedReplicaDir)
    os.Exit(code)
}

Each test cleans the shared directory before running:

t.Run("Simple", func(t *testing.T) {
    // Clean shared directory for this test
    os.RemoveAll(sharedReplicaDir)
    os.MkdirAll(sharedReplicaDir, 0755)

    // Use shared replica directory
    client := file.NewReplicaClient(sharedReplicaDir)

    // ... rest of test uses the VFS registered in TestMain
    db, err := sql.Open("sqlite3", "file:/tmp/test.db?vfs=litestream&mode=ro")
    // Works!
})

Test Results

Both tests pass! 🎉

=== RUN   TestVFS_Integration
=== RUN   TestVFS_Integration/Simple
--- PASS: TestVFS_Integration/Simple (0.21s)
=== RUN   TestVFS_Integration/Updating
--- PASS: TestVFS_Integration/Updating (0.71s)
--- PASS: TestVFS_Integration (0.92s)
PASS
ok      github.com/benbjohnson/litestream/cmd/litestream-vfs   1.444s

Files Changed

Just 3 files showing the working solution:

1. cmd/litestream-vfs/main_test.go

   • Added TestMain() for shared VFS setup
   • Both tests use shared directory approach
   • Clean, simple implementation

2. cmd/litestream-vfs/no_loadable_ext_test.go

   • Build tag helper (unchanged from base)

3. VFS_EXTENSION_FINDINGS.md

   • Complete documentation of investigation
   • Explains why this approach works
   • Shows test results and benefits

Benefits

✅ No environment variable propagation issues
✅ No segfaults (using static linking)
✅ Simple, clean code
✅ Tests run fast
✅ Works exactly as you wanted

Full Details

See VFS_EXTENSION_FINDINGS.md for:

• Complete investigation findings
• Why loadable extensions don't work with sqlite3vfs
• Detailed explanation of the working solution
• Test output and analysis

Ready to Merge

This proves the concept works perfectly with your shared directory constraint. Simple, clean, and passing tests!

[corylanou]

✅ Working Solution Implemented!

Ben's clarification was the key: "If I can get the env vars set once then that's good enough. I don't mind using a single temp directory to run tests against."

What This Means

With a shared temp directory approach, we don't need the complex LazyReplicaClient pattern or extension loading for tests. We can use simple static linking with TestMain setup.

Implementation

Added TestMain() that:

1. Creates shared temp directory (/tmp/litestream-vfs-test-shared)
2. Sets up VFS once for all tests
3. Registers VFS at package level
4. Each test cleans the directory before running

Test Results

Both tests now pass! 🎉

=== RUN   TestVFS_Integration
=== RUN   TestVFS_Integration/Simple
--- PASS: TestVFS_Integration/Simple (0.21s)
=== RUN   TestVFS_Integration/Updating
--- PASS: TestVFS_Integration/Updating (0.71s)
--- PASS: TestVFS_Integration (0.92s)
PASS
ok      github.com/benbjohnson/litestream/cmd/litestream-vfs   1.444s

What Changed

cmd/litestream-vfs/main_test.go:

• Added TestMain() for one-time VFS setup
• Simplified both test functions to use shared directory
• Removed complex openWithVFS() extension loading code
• Much cleaner and easier to understand

VFS_EXTENSION_FINDINGS.md:

• Added "Working Solution" section with implementation details
• Documents the TestMain approach
• Shows test results
• Explains benefits

Benefits

✅ No env var propagation issues (set once in TestMain)
✅ No segfaults (using static linking)
✅ Simple, clean code
✅ Tests run fast
✅ Works exactly as Ben wanted

Note on LazyReplicaClient

The LazyReplicaClient implementation in litestream-vfs.go remains for potential future loadable extension use cases, but is not needed for static linking tests.

For testing purposes, this simpler approach is perfect. If Ben wants to pursue loadable extensions in the future for production use, that would still require solving the sqlite3vfs compatibility issue (likely a pure C VFS implementation).

Next Steps

Ready for Ben to review! This proves the concept works with his constraint.

The tests are passing, the code is clean, and it's ready to integrate.

[corylanou]

🧹 Cleaned Up PR - Ready for Review

I've simplified the branch to show only what's needed for the working solution. Removed all the loadable extension investigation code.

What's Left in This PR

Just 3 essential changes:

1. main_test.go - TestMain() with shared directory approach
2. no_loadable_ext_test.go - Small build tag helper
3. VFS_EXTENSION_FINDINGS.md - Documentation

What Was Removed

• ❌ third_party/sqlite3vfs/ - 32 vendored files (not needed!)
• ❌ cmd/litestream-vfs/loadable_ext_test.go - Loadable extension build tags
• ❌ src/sqlite3-binding.h - 13,358 line header (not needed!)
• ❌ LazyReplicaClient code - Reverted to base branch version
• ❌ All C wrapper modifications - Reverted to base branch
• ❌ Header file changes - Reverted to base branch

Diff Summary

31 files changed, 13813 insertions(+), 28250 deletions(-)

Most of that is deletions - removing unnecessary investigation code!

What You'll See

The PR now cleanly shows:

• How TestMain sets up VFS once with shared directory
• Both tests using the simple approach
• Clear documentation of findings

No complex extension loading, no vendored packages, no LazyReplicaClient - just the simple working solution!

Tests still pass:

--- PASS: TestVFS_Integration/Simple (0.21s)
--- PASS: TestVFS_Integration/Updating (0.71s)
PASS

Ready for your review! 🎉