Improving System Architecture Through Production Issue Analysis

Production Issues in a Data Lakehouse. Adopting an open table format like Apache Hudi, Apache Iceberg or Delta Lake is a crucial first step in moving toward a modular, interoperable data architecture. But once your data starts landing in cloud object stores like S3, GCS, or Azure Blob, real production challenges begin to emerge. Let’s walk through some common pitfalls that teams hit in the wild: ❌ Small File problem: With certain workloads (such as streaming), you may want to write data as soon as it arrives in smaller batches. This can lead to a lot of small files, which ultimately impacts read performance. ❌ Object Store Throttle: In cloud storage systems such as AWS S3, the extensive volume of file listing requests occasionally leads to throttling because of specific request limitations. ❌ Data Co-locality: In analytical workloads, the mismatch between arrival and event time poses a challenge. Writing data quickly often means using arrival time, but this can cause query issues due to data being spread across files. ❌ Long-Tail Partitions Aging Poorly: Older partitions often escape rewrite/compaction cycles, becoming cluttered with small or fragmented files, hurting cold-query performance. ❌ Metadata Management Overhead: As table sizes grow (millions of files, thousands of partitions), query planning suffers, especially without metadata pruning, column stats, or index support. ❌ Concurrency and Job Failures: Race conditions in concurrent writers, snapshot commits, or read-after-write consistency issues surface unless there are robust methods for isolation. ❌ Operational Overhead on Optimization Jobs: Compaction, clustering, snapshot expiration, and vacuum jobs need careful scheduling (if not automatic), otherwise they interfere with query performance or conflict with writes. To run a reliable lakehouse at scale, table formats alone aren’t enough. You need supporting services that handle these operational realities: ✅ Compaction & clustering to optimize storage layout ✅ Partition pruning & metadata indexing to reduce scan overhead ✅ Schema enforcement and evolution tracking ✅ Concurrency control & snapshot isolation ✅ Background services to automate cleanups, retention, and rewrites Lakehouse platforms like Apache Hudi brings various table management services (async/inline) that enables you to natively deal with these issues. Other formats like Iceberg outsources scheduling these to compute engines and needs careful consideration for scheduling. Detailed reading in comments. #dataengineering #softwareengineering

Recently I witnessed a perfect example of how different teams can handle the same type of problem. An internal payment system couldn't get event statuses because another team's service was burning through a shared API rate limit. Instead of 4 calls per day as recommended, they were making erratic calls through inefficient polling. The fix? Simple. Change from continuous polling to the recommended schedule. Maybe 60 minutes of work for one engineer, even without extensive contextual knowledge. Their response? "Please submit this as an Aha idea for our next planning cycle." Meanwhile, at Aurora, we handle operational issues completely differently. When we discover inefficient processes affecting system performance, we: Immediately assess business impact and technical scope Deploy hotfixes within hours, not planning cycles Document the fix and implement monitoring to prevent recurrence Save formal process for actual feature development This approach has helped us maintain 4 9s of uptime on critical data pipelines while other organizations wait for roadmap discussions. The difference? We treat production issues as operational incidents, not feature requests. When systems break or perform poorly, rapid resolution takes priority over process compliance. Good organizations distinguish between different types of work: Feature development follows formal product processes with discovery and planning Operational issues get fast-tracked through engineering channels with immediate triage Performance optimizations get handled as technical debt within existing sprint capacity At Aurora, our leadership empowers our teams to make quick decisions on operational issues while maintaining appropriate governance for new features. It's one of the things that makes working here effective and why I love leading our Enterprise Data & Automation practice. The result? Our data systems stay reliable, our business operations stay smooth, and we can focus planning cycles on actual innovation instead of firefighting. Process should enable better outcomes, not slow them down. The best teams know when to follow process and when to cut through it. What does rapid operational response look like at your organization?

USING FMEA CONCEPTS IN SITE RELIABILITY ENGINEERING (SRE) In Site Reliability Engineering, the goal isn’t just to respond to failures — it’s to reduce the chance of failures happening in the first place. One framework that aligns very naturally with this mindset is Failure Mode and Effects Analysis (FMEA). How FMEA translates into SRE practices Failure modes Start by asking a simple question: How can this system fail? Examples include pod crash loops, node pressure issues, API latency spikes, database connection exhaustion, or autoscaling misconfigurations. Effects analysis Evaluate the impact of each failure: • Customer-facing outages • Latency degradation • Error budget burn • SLA or SLO violations Prioritization Not all failures deserve equal attention. Reliability work is prioritized based on: • Severity of impact • Likelihood of occurrence • Difficulty of detection Controls and mitigations Turn analysis into action through: • Alerts aligned with SLIs • Autoscaling and graceful degradation • Load shedding and circuit breakers • Runbooks and automated remediation • Controlled failure testing Why this matters Applying FMEA principles helps teams move from reactive incident handling to proactive reliability engineering. It supports better error budget management, reduces MTTR, and builds a shared understanding of system risk. Key takeaway Most production incidents are known failure modes that were never formally examined. FMEA provides a structured way to surface these risks before users are impacted. #SRE #SiteReliabilityEngineering #FMEA #DevOps #CloudEngineering #Reliability #Observability #IncidentManagement

Over six months, a staff engineer tracked the origins of several production incidents. The findings were awkward. Ten potential failures were avoided before any code was written. And yet, the reasons were neither testing nor monitoring. It was a weekly architecture meeting widely regarded as a formality, often seen as a waste of time. The arithmetic was unglamorous but clear. Ninety minutes a week for eight engineers comes to roughly $100,000 a year. Judged by likely revenue loss, engineering hours and customer fallout, the avoided incidents cost came to about $500,000. A meeting dismissed as overhead was returning five dollars for every one spent. Three incidents did occur during the period, and two happened in weeks when the meeting was dropped on the grounds that “nothing much was happening.” The episode highlights a familiar pattern. Companies tend to prune coordination first and restore it only after the resulting disorder becomes expensive. The better ones pay attention to what did not happen—and why.

Most RAG systems fail in production. Not because of models, but because of system design. A lot of teams think building a RAG system is simple: • Chunk data • Create embeddings • Store in a vector database • Retrieve and generate That is only the starting layer. The real complexity begins after that. Looking at production-grade systems, the architecture is much deeper. It starts with data ingestion: • Clean, structured chunking • Context-aware preprocessing Then embeddings: • Dense for semantic understanding • Sparse for keyword precision Then storage: • Choosing the right vector database based on scale and latency But this is still not enough. The real differentiation comes in retrieval: • Exact retrieval for precision • Semantic retrieval for context • Hybrid approaches for balance And most systems break right here. Because retrieval quality directly impacts output reliability. That is why evaluation becomes critical: • Retrieval quality metrics • Generation accuracy • Online and offline evaluation • LLM as a judge for validation Without this, you are guessing, not engineering. Then comes the advanced layer: • Re-ranking to improve relevance • Graph-based retrieval for relationships • Agentic RAG for multi-step reasoning This is where systems become intelligent. But even this is not the final stage. What actually makes a system production-ready is: Observability: • Logging and traceability • Key performance metrics • Drift detection over time Production engineering: • Latency optimization • Cost control • Failure handling strategies This is the part most people ignore. And this is exactly why many AI projects never move beyond demos. From my experience, the gap is always the same: • Not models. • Not tools. • But system thinking. If you are building RAG systems, the goal is not to make them work once. The goal is to make them reliable, measurable, and scalable. If you are currently working on a RAG or AI system and facing performance, cost, or reliability issues, feel free to share your use case. Happy to give a direct perspective. About Me I help SaaS and enterprise teams move from AI pilots to production-grade systems that deliver measurable outcomes. Most AI initiatives fail not because of models, but due to weak orchestration, poor context handling, and uncontrolled costs. That is where I focus. I design AI systems that: • Automate multi-step workflows across CRMs, ERPs, and internal tools • Ground decisions using structured and agentic RAG • Orchestrate tasks using multi-agent systems • Optimize latency, performance, and cost • Integrate seamlessly with real-world APIs and platforms This leads to: → Lower operational overhead → Faster execution across teams → Reduced manual coordination → Controlled and predictable AI cost I have delivered production AI systems across industries and work at the top of AI consulting. If you are serious about deploying AI into real workflows, let’s connect. 🔗 Upwork: https://lnkd.in/e78fNHex