Benefits of Caching Techniques

𝗡𝗼𝘁 𝗲𝘃𝗲𝗿𝘆 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 𝗻𝗲𝗲𝗱𝘀 𝗮 𝗯𝗶𝗴𝗴𝗲𝗿 𝗲𝗻𝗴𝗶𝗻𝗲. 𝗦𝗼𝗺𝗲𝘁𝗶𝗺𝗲𝘀 𝗶𝘁 𝗷𝘂𝘀𝘁 𝗻𝗲𝗲𝗱𝘀 𝗮 𝘀𝗺𝗮𝗿𝘁𝗲𝗿 𝗽𝗹𝗮𝗰𝗲 𝘁𝗼 𝗽𝘂𝘁 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿. Caching is the ultimate act of compute avoidance — placing pre-computed or frequently accessed data closer to where it's consumed, so the system doesn't repeat expensive work on every request. 𝗪𝗵𝗲𝗿𝗲 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝗹𝗶𝘃𝗲𝘀 𝗶𝗻 𝗱𝗮𝘁𝗮 𝘀𝘆𝘀𝘁𝗲𝗺𝘀: → 𝗤𝘂𝗲𝗿𝘆 𝗰𝗮𝗰𝗵𝗲: Database stores results of recent queries. Same query, same parameters? Serve cached result instead of re-scanning. → 𝗠𝗮𝘁𝗲𝗿𝗶𝗮𝗹𝗶𝘇𝗲𝗱 𝘃𝗶𝗲𝘄𝘀: Pre-computed query results that refresh on a schedule. Caching at the SQL layer fast reads from a pre-built table instead of real-time JOINs. → 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗰𝗮𝗰𝗵𝗲: Redis, Memcached, or in-memory stores between the app and database. Reduces database load for hot-path lookups like session data, catalogs, or feature flags. → 𝗖𝗗𝗡 / 𝗲𝗱𝗴𝗲 𝗰𝗮𝗰𝗵𝗲: Content served from locations closer to the user. Relevant for serving dashboards and reports at scale. → 𝗦𝗽𝗮𝗿𝗸 𝗰𝗮𝗰𝗵𝗲: .cache() or .persist() to keep intermediate DataFrames in memory across stages. Avoids recomputing the same transformation in multi-step pipelines. 𝗧𝗵𝗲 𝘁𝗿𝗮𝗱𝗲-𝗼𝗳𝗳 𝘁𝗵𝗮𝘁 𝗻𝗲𝘃𝗲𝗿 𝗰𝗵𝗮𝗻𝗴𝗲𝘀: Speed vs staleness. Every cache is a snapshot of a past state. The faster it serves, the more likely it's serving data that's no longer current. Cache invalidation remains one of the hardest problems not because it's complex to code, but because it's complex to get right under changing data. 𝗧𝗵𝗲 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗿𝘂𝗹𝗲: Cache what's expensive to compute, frequently accessed, and tolerant of slight staleness. If freshness is non-negotiable, caching is a liability not a shortcut. Where in your stack are you trading freshness for speed and is the trade-off still worth it? #DataEngineering #DataArchitecture #SystemDesign

You might think “caching” = Redis. But in real system design… Caching is a stack, not a single layer. Different caches live in different places, solve different problems, and break in different ways. Here are 8 types of caching you’ll actually use in system design 👇 1) Browser Cache The first cache layer - stores static frontend files in the user’s browser so repeat visits feel instant. 2) CDN Cache Caches images/videos/JS/CSS at edge locations worldwide, reducing latency and protecting the origin from traffic spikes. 3) Reverse Proxy Cache Sits between client and backend (NGINX/Varnish) to cache API responses/pages and reduce backend load. 4) Application Cache Lives inside your service layer - caches computed results, user sessions, feature flags, and frequent query outputs. 5) Database Cache Caches query results / hot rows near the DB layer to reduce DB I/O and speed up repeated reads. 6) Distributed Cache A shared cache layer (Redis/Memcached) used across services - essential for microservices and horizontal scaling. 7) Write-Through Cache Writes go to cache + DB together - best for strong consistency where stale data is unacceptable. 8) Write-Back Cache (Write-Behind) Writes go to cache first, DB later asynchronously - best for high-write systems, but needs durability + recovery planning. ✅ If you understand these 8 cache types… you can design systems that are fast, scalable, and stable under load.

Uber's database handles 40M reads/s. 4.1 ms at p99.9 latency explained: Uber's Docstore is a massive, distributed database essential for handling the company's data needs, processing more than 30 million requests per second. As demand grew, the need for low-latency, high-throughput solutions became critical. Traditional disk-based storage, even with optimizations like NVMe SSDs, faced limitations in scalability, cost, and latency. To address these challenges, Uber developed CacheFront, an integrated caching solution designed to reduce latency, improve scalability, and lower costs without compromising on data consistency or developer productivity. The Key Challenges: 0. Latency and scalability: Disk-based databases have inherent latency and scalability limits. 1. Cost: Scaling up or horizontally adds significant costs. 2. Operational complexity: Managing increased partitions and ensuring data durability is complex. 3. Request imbalance: High-read request volumes can overwhelm storage nodes. CacheFront's solution had to implement cached reads, with disk fallback if necessary. It also had to handle cache invalidation with a data capture system and remain adaptable per database, table, or request basis. Implementation Highlights: 0. Incremental build: Started with the most common query patterns for caching. 1. High-level architecture: Separates caching from storage, allowing independent scaling. 2. Cache invalidation: Utilizes change data capture to maintain consistency. 3. Cache warming and sharding: Ensures high availability and fault tolerance across geographical regions. 4. Circuit breakers and adaptive timeouts: Enhances system resilience and optimizes latency. Results and Impact: 0. P75 latency decreased by 75%, and P99.9 latency by over 67%. 1. Achieved a 99% cache hit rate for some of the largest use cases, significantly reducing the load on the storage engine. 2. Reduced the need for approximately 60K CPU cores to just 3K Redis cores for certain use cases. 3. Supports over 40 million requests per second across all instances, with proven success in failover scenarios. Actionable Learnings: 0. Integrated caching: Implementing an integrated caching layer can dramatically improve database read performance while reducing costs and operational complexity. 1. Cache invalidation: A robust cache invalidation strategy is crucial for maintaining data consistency, especially in systems requiring high throughput and low-latency reads. 2. Adaptability and scalability: Systems should be designed to adapt to varying workloads and scale independently across different components to ensure reliability and performance. What do you think about it? #softwareengineering #systemdesign #programming

No Caching = Performance Bottleneck One of the most overlooked cloud performance antipatterns is not caching data at all. You’d be surprised how many systems fetch the same data repeatedly—despite it rarely changing. Here’s what happens when you fall into the No Caching Antipattern: 🔁 Repeated DB queries for identical data 🐌 Slow response times under load 🔥 Increased I/O, latency, and cloud costs ⛔️ Risk of service throttling or failure ✅ The Fix? Cache-Aside Pattern 1. Try to get from cache 2. If not found, fetch from DB + store in cache 3. Invalidate or update on write How to Detect the No Caching Antipattern 🔍 Review app design: Is any cache layer used? Which data changes slowly? 📊 Instrument the system: How often are the same requests made? 🧪 Profile the app: Check I/O, CPU, and memory usage in a test environment 🚦 Load test: Simulate realistic workloads to measure impact under stress 📈 Analyze DB/query stats: Which queries are repeated the most? Tip: Even if data is volatile or short-lived, smart caching strategies (with TTL, invalidation, and fallbacks) can massively improve resilience and scalability. Cache wisely. Profile constantly. Monitor cache hit rates. Because not caching is costing you more than you think. Have you encountered this in the wild? Drop your experience below 👇

If your agent doesn't cache, it's burning budget! Traditional LLM queries behaves predictably: you ask once, wait briefly, and get a clear answer. But AI agents think differently - they plan, retry, call external tools, and loop through multiple reasoning steps. This can quickly blow up latency and costs. Here's the three ways caching saves your AI agents from cost meltdown. 1. Mem caching: Keep a simple in-session storage that remembers what your agent already asked or computed, preventing duplicated calls. It holds intermediate tool results, retrieved snippets, or reasoning steps so the agent doesn’t redo work inside the same task. 2. Prefix caching: Break agent answers into chunks. Save reusable pieces so subsequent queries just extend from a known prefix. If the agent already generated the first 400 tokens of a long answer, you can resume from there instead of re‑prompting the whole sequence. 3. Prompt caching: Set up a basic store that maps frequent prompts directly to answers, skipping repeated computation for identical questions. Great for FAQs and idempotent calls; useless when the context changes per request. Caching is about keeping your agents responsive and your costs predictable. It’s how you turn clever loops into predictable, affordable workflows. Keep shining a light on it - most teams still overlook the basics. #FinOps #Mavvrik #AgenticAI

𝗥𝗔𝗚 𝗶𝘀 𝗚𝗼𝗻𝗲, 𝗟𝗼𝗻𝗴 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗪𝗶𝗻𝗱𝗼𝘄𝘀 𝗮𝗿𝗲 𝗛𝗲𝗿𝗲! A while ago, everyone was hyped about 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹-𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿��𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚). It seemed like the go-to solution for handling large documents and enhancing context in AI applications. But now, things are changing. Long context window models are proving to be game-changers! Instead of chunking documents, these models can handle entire documents, maintaining context and accuracy, especially for summarization tasks. 𝗪𝗵𝗮𝘁 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗰𝗼𝘀𝘁𝘀? Initially, you might think that processing long inputs would be costly. However, there's a fascinating concept called input caching. Input caching allows you to store and reuse repeated input tokens, significantly reducing the cost. 𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗶𝗻𝗽𝘂𝘁 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝘄𝗼𝗿𝗸? Whenever we have a Q&A problem on the same context, we pass the context along with the user's question to the model. While every user's query might be different, the context remains the same. This leads to repeated tokens in the input. Instead of reprocessing these repeated tokens every time, input caching stores them. On subsequent uses, the model simply retrieves these tokens from the cache instead of reprocessing them. This not only reduces the computational load but also significantly cuts down on the cost. 𝗪𝗵𝘆 𝗱𝗼𝗲𝘀 𝗶𝗻𝗽𝘂𝘁 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝗺𝗮𝗸𝗲 𝘀𝗲𝗻𝘀𝗲? When dealing with scenarios where the context is static but questions vary, input caching shines. By caching the repeated input tokens and their calculations, the model can focus computational resources on processing the new questions rather than re-evaluating the same context repeatedly. This speeds up processing time and lowers costs. 𝗦𝗼𝘂𝗿𝗰𝗲𝘀 : https://lnkd.in/gUvrU5nS

When RAG isn’t enough and CAG becomes essential Imagine you’re building a voice assistant for aircraft maintenance engineers. Each aircraft type has tens of thousands of validated maintenance procedures, torque settings, inspection steps, fault codes stored in secure internal manuals that rarely change. At first glance, you might think of using Retrieval-Augmented Generation (RAG). Every time an engineer asks, “What’s the torque setting for the A320 hydraulic pump valve?” the system would: Retrieve relevant text from thousands of manuals, Rank and embed them, Then feed that data into the LLM for response generation. This works beautifully when knowledge is dynamic and expansive ,but in this case, it introduces unnecessary latency, indexing dependencies, and retrieval noise, even though the knowledge is largely static and validated. Enter Cache-Augmented Generation (CAG) Now, imagine replacing those repeated retrieval calls with a cached knowledge layer. With CAG, frequently used and stable information, like standard procedures or torque settings ,is pre-encoded and cached in a key-value memory or persistent context store. When the engineer asks a question, the LLM doesn’t perform an external retrieval. It fetches from the cache, instantly pulling the pre-validated answer context into generation. Why CAG makes sense here The domain is bounded and stable , knowledge doesn’t change often Speed and reliability matter more than continuous updates Only a high value subset of knowledge needs caching, not all 50,000 documents The system still supports retrieval as fallback for new or un-cached data So, the assistant now behaves intelligently: It fetches known procedures directly from cache, It retrieves only when something new or unknown is asked. My take : Hybrid is the future No enterprise runs purely on CAG or RAG alone. Modern architectures use adaptive hybrids , cache for repeatable, validated knowledge; retrieval for evolving information. This hybridization reduces latency, maintains freshness, and provides deterministic accuracy where it matters most. #RAG #CAG #AI #LLM's Image credit Mohamad Hassan, unable to tag him.

You can reduce latency by >2x and costs up to 90% with "prompt caching". Prompt Caching enables caching of frequently used context between API calls — resulting with shorter response times, and lower processing costs. But how does it work? When an LLM processes a prompt, it generates internal representations called attention states, which help the model understand relationships between different parts of the input. Traditionally, these attention states are recalculated every time the model processes a prompt, even if the input is similar or repeated, making it time-consuming and costly. Prompt caching solves this by storing previously computed attention states, allowing the model to retrieve them for similar prompts, speeding up responses and reducing costs. So, when should you use it? Prompt caching is useful for sending large prompt context once and reusing it in future requests, reducing cost and latency. 1/ In conversational agents, it enhances extended dialogues by avoiding repeated input of long instructions. 2/ For coding assistants, it helps by keeping a summarized version of the codebase in the prompt for faster autocomplete and Q&A. 3/ In large document processing, it allows full documents or images to be included without slowing down response times. 4/ It also improves the quality of responses in detailed instruction sets and agentic search by maintaining a cache of multiple examples and iterative tool calls. You're probably using this feature automatically with OpenAI models, and you can set it up manually for Anthropic and Gemini models. Check our parameter library for more info: https://lnkd.in/dczSJ6yA