Weaviate 1.36 is here! 🔥 HNSW is the gold standard for vector search, but it needs everything in memory. As datasets grow, that gets expensive. HFresh takes a different approach: partition vectors into disk postings, keep only a small centroid index in memory, and search by reading just the relevant slices from disk. The result: significantly lower memory usage with predictable latency, even at billions of vectors. No periodic rebuilds — the index stays fresh through incremental background rebalancing. Also in this release, five features are becoming generally available: - Server-side Batching — the server controls ingestion flow, no more manual batch tuning - Object TTL — automatic expiration with flexible strategies (creation time, update time, or date property) - Async Replication — now with per-collection configuration - Drop Inverted Indices — reclaim disk space by removing unused indices - Backup Restoration Cancellation — cancel in-progress restores Full details in the blog post: https://lnkd.in/ej3mQbSC
Weaviate’s Post
More Relevant Posts
-
Today Hugging Face are introducing HF-MOUNT 🔥 → Mount any repo from the Hub (Storage Buckets, but also models, datasets) as a local filesystem This is a game changer, as it allows you to attach remote storage that is 100x bigger than your local machine's disk. Read-write for Storage Buckets, read-only for models and datasets. Here's an example with FineWeb-edu (a 5TB slice of the Web): 1️⃣> hf-mount start repo datasets/HuggingFaceFW/fineweb-edu /tmp/fineweb It takes a few seconds to mount, and then: 2️⃣> du -h -d1 /tmp/fineweb 4.1T ./data 1.2T ./sample 5.3T . Two backends are available: NFS (recommended) and FUSE 👉 Follow Ultan O. for more
To view or add a comment, sign in
-
-
Live A/B Test: SULCUS vs. Standard Context Currently working with two OpenClaw instances running Claude Sonnet 4.6 in parallel to validate the operational impact of a Virtual Memory Management Unit (vMMU). Instance A: Standard deployment. Instance B: Integrated with the SULCUS sidecar. Initial Observations (T+ several hours): State Retention: Instance A is exhibiting standard context decay (struggling to recall early session parameters). Instance B is maintaining stable relational context without truncation. Latency: The SULCUS-backed instance is currently demonstrating faster response times. Need to validate the exact telemetry on the latency improvement. Early data suggests that actively paging out "cold" data via the vMMU reduces the processing overhead on the active prompt, speeding up inference. Will publish the raw logs once the test concludes.
To view or add a comment, sign in
-
Join Vinicius and Guilherme on learn more about our latest feature that is helping customers save thousands while still reducing the infrastructure costs.
Most production databases handle operational and analytical queries from the same system. Deep Cache was built for the mission-critical ones. Shallow Cache handles the rest. We're hosting a live walkthrough of Readyset Shallow Cache on April 7. Near 100% query support, zero ramp-up time, and no rewrites required. If you're managing database performance at scale, come take a look. Register here 👉 https://lnkd.in/g5eXnfFz
To view or add a comment, sign in
-
TidesDB v8.7.0 MINOR has been released. This release focuses on optimizing iterator and seek performance with block reuse fast paths, cached memtable sources, and a new raw byte block cache. Block manager writes now use pwritev for reduced syscalls. Includes various correctness fixes and new APIs for database stats, column family purging, and a selective test runner. https://lnkd.in/ewX9bhnm
To view or add a comment, sign in
-
How much memory does it take to store 1 million documents in a RAG system? Cheat sheet covers: The storage formula How embedding dimension affects size Chunking strategy tradeoffs Full worked example (1M docs) The infrastructure stack you actually need Hope it helps someone 🙂
To view or add a comment, sign in
-
DBNL is a powerful layer in your production agent stack. See how it works with a simple calculator agent example: https://dbnl.io/uYWVjB
To view or add a comment, sign in
-
-
Killer feature for your product: add server-side caching with at least a 3600-second TTL, and revalidate it on core events (e.g., when the description or price is updated) But never rely on frequent revalidation for large datasets—keep them eventually consistent instead
To view or add a comment, sign in
-
We just released #QuestDB 9.3.4. This release introduces dynamic window support in WINDOW JOIN, allowing window ranges to be computed from column values or expressions, which makes many time-series analytics queries much more flexible. We also added Parquet row group pruning with bloom filters, reducing I/O significantly for selective queries over Parquet data. Other highlights include: • New array functions for DOUBLE[] such as sort, reverse, min, max, avg, and sum • Performance improvements across ORDER BY, ASOF JOIN, WINDOW JOIN, HORIZON JOIN, GROUP BY, and Parquet I/O • Improved Parquet writing and decoding • WAL writer tuning options for memory access patterns Overall, this release continues to improve both analytical capabilities and performance across the engine. Link in first comment 👇
To view or add a comment, sign in
-
✅ Best Practices to Reduce IO_COMPLETION Waits in #sqlserver 1. Optimize queries: Reduce sort operations, add indexes to avoid large spills. 2. Increase #memory (RAM): Helps prevent sort spills to disk. 3. Separate #transaction log and data files: Place logs on fast, dedicated disks. 4. Use SSDs or faster #storage: Lower #latency reduces #synchronous I/O waits. 5. Monitor #backups/ #restores: Schedule during off-peak hours to avoid contention. 6. Check disk #subsystem health: Use tools like SQLIO or DiskSpd to benchmark.
To view or add a comment, sign in
-
-
Stop using async def for everything in FastAPI. You’re likely killing your performance without realizing it. Just because you can make an endpoint asynchronous doesn't mean you should. If you call a blocking function (like a standard DB query or time.sleep) inside an async block, you freeze the entire Event Loop. Swipe through to see: When to use def vs async def. How the Event Loop vs. Thread Pool actually works.
To view or add a comment, sign in
Weaviate•24K followers
4wNice! This is a big one!