Best Practices for Deploying LLM Systems

Few Lessons from Deploying and Using LLMs in Production Deploying LLMs can feel like hiring a hyperactive genius intern—they dazzle users while potentially draining your API budget. Here are some insights I’ve gathered: 1. “Cheap” is a Lie You Tell Yourself: Cloud costs per call may seem low, but the overall expense of an LLM-based system can skyrocket. Fixes: - Cache repetitive queries: Users ask the same thing at least 100x/day - Gatekeep: Use cheap classifiers (BERT) to filter “easy” requests. Let LLMs handle only the complex 10% and your current systems handle the remaining 90%. - Quantize your models: Shrink LLMs to run on cheaper hardware without massive accuracy drops - Asynchronously build your caches — Pre-generate common responses before they’re requested or gracefully fail the first time a query comes and cache for the next time. 2. Guard Against Model Hallucinations: Sometimes, models express answers with such confidence that distinguishing fact from fiction becomes challenging, even for human reviewers. Fixes: - Use RAG - Just a fancy way of saying to provide your model the knowledge it requires in the prompt itself by querying some database based on semantic matches with the query. - Guardrails: Validate outputs using regex or cross-encoders to establish a clear decision boundary between the query and the LLM’s response. 3. The best LLM is often a discriminative model: You don’t always need a full LLM. Consider knowledge distillation: use a large LLM to label your data and then train a smaller, discriminative model that performs similarly at a much lower cost. 4. It's not about the model, it is about the data on which it is trained: A smaller LLM might struggle with specialized domain data—that’s normal. Fine-tune your model on your specific data set by starting with parameter-efficient methods (like LoRA or Adapters) and using synthetic data generation to bootstrap training. 5. Prompts are the new Features: Prompts are the new features in your system. Version them, run A/B tests, and continuously refine using online experiments. Consider bandit algorithms to automatically promote the best-performing variants. What do you think? Have I missed anything? I’d love to hear your “I survived LLM prod” stories in the comments!

If I had to make LLM systems reliable in production, I wouldn’t start by adding more prompts. I’d focus on mastering these ideas: • Grounding outputs back to source data • Designing clear input and output contracts • Detecting when the model is uncertain • Validating structured outputs before use • Isolating failures so one bad call doesn’t break the system • Adding checkpoints instead of long fragile chains • Building retries with intent, not blind loops • Logging decisions, not just final answers • Evaluating behavior over time, not one-off responses None of this shows up in demos. All of it shows up in real systems. Most LLM failures aren’t “model issues”. They’re engineering discipline issues. If you care about deploying GenAI beyond notebooks, these are the skills that actually matter. #LLM #GenAI #AIEngineering #ProductionAI #SystemsDesign #Interviews #AI #Jobs Follow Sneha Vijaykumar for more... 😊

I recently spent time getting more hands-on with LLM & Agentic AI engineering through Ed Donner's training. Instead of stopping at examples, I built a mini multi-agent logistics delivery optimization framework. Building real AI systems quickly makes one thing clear: 𝙏𝙝𝙚 𝙝𝙖𝙧𝙙 𝙥𝙖𝙧𝙩 𝙞𝙨𝙣’𝙩 𝙩𝙝𝙚 𝙢𝙤𝙙𝙚𝙡 — 𝙞𝙩’𝙨 𝙩𝙝𝙚 𝙖𝙧𝙘𝙝𝙞𝙩𝙚𝙘𝙩𝙪𝙧𝙚 𝙙𝙚𝙘𝙞𝙨𝙞𝙤𝙣𝙨 𝙖𝙧𝙤𝙪𝙣𝙙 𝙞𝙩. A few practical lessons: 1. 𝗟𝗟𝗠 𝗺𝗼𝗱𝗲𝗹 𝘀𝗲𝗹𝗲𝗰𝘁𝗶𝗼𝗻 𝗶𝘀 𝗳𝗮𝗿 𝗺𝗼𝗿𝗲 𝗻𝘂𝗮𝗻𝗰𝗲𝗱 𝘁𝗵𝗮𝗻 𝗰𝗼𝘀𝘁 𝘃𝘀 𝗹𝗮𝘁𝗲𝗻𝗰𝘆. Trade-offs: • reasoning maturity for complex planning • context window & memory strategy • proprietary models vs smaller open models • infra costs (GPU/hosting) vs token-based API costs • tool-calling reliability & structured output adherence • benchmark performance vs real task behavior • model stability across releases In practice, it becomes a hybrid strategy: 𝘀𝗺𝗮𝗹𝗹𝗲𝗿/𝗰𝗵𝗲𝗮𝗽𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀 𝗳𝗼𝗿 𝗿𝗼𝘂𝘁𝗶𝗻𝗲 𝘁𝗮𝘀𝗸𝘀 + 𝗦𝗟𝗠 𝘄𝗶𝘁𝗵 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗳𝗼𝗿 𝗱𝗼𝗺𝗮𝗶𝗻 𝗽𝗿𝗼𝗯𝗹𝗲𝗺𝘀 + 𝘀𝘁𝗿𝗼𝗻𝗴𝗲𝗿 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹𝘀 𝗳𝗼𝗿 𝗰𝗼𝗺𝗽𝗹𝗲𝘅 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀. 𝟮. 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗮𝘀 𝗺𝘂𝗰𝗵 𝗮𝘀 𝘁𝗵𝗲 𝗟𝗟𝗠: Many AI demos over-engineer the stack. In reality, simplicity, latency, security and reliability matter more than novelty. • Use orchestration frameworks only where coordination complexity exists • Combine prompts with structured outputs to reduce ambiguity • Watch serialization and tool-call overhead — they impact latency and UX • Reduce unnecessary LLM calls when deterministic code can solve the task Besides lowering token cost, this improves context efficiency, letting models focus on real reasoning. Sometimes best architecture decision is 𝙣𝙤𝙩 𝙞𝙣𝙩𝙧𝙤𝙙𝙪𝙘𝙞𝙣𝙜 𝙖𝙣𝙤𝙩𝙝𝙚𝙧 𝙡𝙖𝙮𝙚𝙧. 3. 𝗕𝗶𝗴𝗴𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀 ≠ 𝗯𝗲𝘁𝘁𝗲𝗿 𝗼𝘂𝘁𝗰𝗼𝗺𝗲𝘀 Smaller models with fine-tuning on domain data can perform more consistently than larger ones. Fine-tuning helps when: • tasks are repetitive but require precision • domain vocabulary is specialized • prompts become fragile But 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗮𝗹𝘀𝗼 𝗶𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗲𝘀 𝗹𝗶𝗳𝗲𝗰𝘆𝗰𝗹𝗲 𝗼𝘃𝗲𝗿𝗵𝗲𝗮𝗱. Base model upgrades trigger retesting and partial rewrites. 4. 𝗧𝗵𝗲 𝗿𝗲𝗮𝗹 𝗴𝗮𝗽: 𝗽𝗿𝗼𝘁𝗼𝘁𝘆𝗽𝗲 → 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 Demos are easy. Production requires 𝙚𝙫𝙖𝙡𝙪𝙖𝙩𝙞𝙤𝙣 𝙛𝙧𝙖𝙢𝙚𝙬𝙤𝙧𝙠𝙨, 𝙤𝙗𝙨𝙚𝙧𝙫𝙖𝙗𝙞𝙡𝙞𝙩𝙮, 𝙨𝙚𝙘𝙪𝙧𝙞𝙩𝙮, 𝙥𝙚𝙧𝙛𝙤𝙧𝙢𝙖𝙣𝙘𝙚, 𝙘𝙤𝙨𝙩 𝙜𝙤𝙫𝙚𝙧𝙣𝙖𝙣𝙘𝙚 & 𝙜𝙪𝙖𝙧𝙙𝙧𝙖𝙞𝙡𝙨. That’s where most engineering effort goes. 𝟱. 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗳𝗼𝗿 𝗹𝗲𝗮𝗱𝗲𝗿𝘀 𝗿𝘂𝗻𝗻𝗶𝗻𝗴 𝗔𝗜 𝗽𝗿𝗼𝗴𝗿𝗮𝗺𝘀 Many AI conversations focus on SDLC productivity- Useful but the bigger opportunity is 𝙧𝙚𝙞𝙢𝙖𝙜𝙞𝙣𝙞𝙣𝙜 𝙡𝙚𝙜𝙖𝙘𝙮 𝙗𝙪𝙨 𝙥𝙧𝙤𝙘𝙚𝙨𝙨𝙚𝙨 𝙪𝙨𝙞𝙣𝙜 𝘼𝙜𝙚𝙣𝙩𝙞𝙘 AI. By simply automating existing steps, we risk making inefficient tasks efficient and missing the real transformation.

What it takes to take AI Agents from prototype to production? After taking multiple AI agents to production, here's what the gap between demo and deployment actually looks like: 𝗦𝗶𝗻𝗴𝗹𝗲-𝗮𝗴𝗲𝗻𝘁 𝗰𝗵𝗮𝗶𝗻𝘀 𝗱𝗼𝗻'𝘁 𝘀𝗰𝗮𝗹𝗲. Linear workflows can't handle failures, recover from rate limits, or maintain state across complex operations. Graph-based architectures give you explicit state management, pause-and-resume capabilities, and failure recovery paths. LangGraph has become the de facto standard here. 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝘀 𝗟𝗟𝗠-𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗰 𝘁𝗼𝗼𝗹𝗶𝗻𝗴. Critical dimensions here include - Was the response grounded? Did retrieval return relevant context? What caused the quality regression? You need platforms that understand token costs, trace agentic workflows, and monitor quality metrics alongside latency. OpenTelemetry provides the foundation, but specialized tools (Langfuse, LangSmith) capture more intricate metrics for LLM systems. 𝗖𝗼𝘀𝘁 𝘄𝗶𝗹𝗹 𝘀𝗽𝗶𝗿𝗮𝗹 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗽𝗿𝗼𝗽𝗲𝗿 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗲𝘀.  1️⃣ Semantic caching delivers 20-30% reduction for repetitive queries.  2️⃣ Model routing sends simple queries to mini models and complex ones to premium. 3️⃣ Prompt compression (using LLMLingua) reduces token usage 15-40% without quality loss. 5️⃣ Batch processing provides automatic 50% discounts for non-urgent work.  The key insight: instrument cost per query from day one and optimize based on usage patterns. 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗺𝘂𝘀𝘁 𝗯𝗲 𝗳𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻𝗮𝗹. Prompt injection remains the top threat. Deploy multi-layered defenses immediately. Guardrails (like NVIDIA NeMo Guardrails) are the first line of defense, filtering malicious inputs and steering conversations. For customer-facing products, PII detection and redaction (using tools like Microsoft Presidio) are essential to prevent data leakage 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀 𝗿𝗲𝗽𝗹𝗮𝗰𝗲 𝘁𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝘁𝗲𝘀𝘁𝗶𝗻𝗴. Unit tests break with non-deterministic outputs. Production systems need RAGAS for retrieval quality, LLM-as-judge for scalable assessment, golden test sets that grow with edge cases, and continuous sampling of production traffic. Set quality gates: if hallucination scores degrade beyond threshold, block deployment. 𝗜𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝘃𝘀 𝗲𝘅𝘁𝗲𝗿𝗻𝗮𝗹 𝗮𝗴𝗲𝗻𝘁𝘀 𝗮𝗿𝗲 𝗳𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹𝗹𝘆 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝘀. Internal tools can iterate with 85% accuracy, known users, and controlled rollout. External products require 95%+ accuracy, handle adversarial inputs, meet compliance requirements (GDPR, SOC2), and provide 99.9% uptime. Development timelines differ by 3-4x. Security needs are entirely different. NotebookLM link in comments below. #ai #agents #llm

Are you getting ready to deploy your LLM app into production?  Here is a practical checklist: First up, Observability. We learned this the hard way - LLM applications can fail silently and in weird ways. Start with basic logging (yes, boring but essential), but if you're building anything complex - like multi-step agents or workflows - you absolutely need tracing. This is exactly why we built OpenLLMetry, our open-source solution based on OpenTelemetry. Next up - make sure to set up some metrics. And I’m not just talking about the obvious stuff - like error rates and latency.  What we've found is that quality metrics are where the real insights hide.  Sometimes even just tracking response length can give you real value. One of our customers for example - caught a major issue just by tracking it. Their LLM suddenly generated much shorter responses. And finally, User Feedback. There are two ways to do this, and you need both.  1- Implicit feedback—Watch what users actually do. Do they commit that auto-generated code? These actions tell you more than any survey ever will. 2 - explicit feedback—simple things like thumbs up or down, quick ratings.  Just make sure you're systematically collecting it. Our open source OpenLLMetry handles all of this out of the box, making it easy from day one. What's your checklist? 

If you’re building anything with LLMs, your system architecture matters more than your prompts. Most people stop at “call the model, get the output.” But LLM-native systems need workflows, blueprints that define how multiple LLM calls interact, how routing, evaluation, memory, tools, or chaining come into play. Here’s a breakdown of 6 core LLM workflows I see in production: 🧠 LLM Augmentation Classic RAG + tools setup. The model augments its own capabilities using: → Retrieval (e.g., from vector DBs) → Tool use (e.g., calculators, APIs) → Memory (short-term or long-term context) 🔗 Prompt Chaining Workflow Sequential reasoning across steps. Each output is validated (pass/fail) → passed to the next model. Great for multi-stage tasks like reasoning, summarizing, translating, and evaluating. 🛣 LLM Routing Workflow Input routed to different models (or prompts) based on the type of task. Example: classification → Q&A → summarization all handled by different call paths. 📊 LLM Parallelization Workflow (Aggregator) Run multiple models/tasks in parallel → aggregate the outputs. Useful for ensembling or sourcing multiple perspectives. 🎼 LLM Parallelization Workflow (Synthesizer) A more orchestrated version with a control layer. Think: multi-agent systems with a conductor + synthesizer to harmonize responses. 🧪 Evaluator–Optimizer Workflow The most underrated architecture. One LLM generates. Another evaluates (pass/fail + feedback). This loop continues until quality thresholds are met. If you’re an AI engineer, don’t just build for single-shot inference. Design workflows that scale, self-correct, and adapt. 📌 Save this visual for your next project architecture review. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

Bigger context windows will not save your LLM app. Most teams think the solution is to stuff more data into the model. It is not. The real advantage comes from Context Engineering. This is the skill of designing an AI system that feeds the model the right information at the right time. Not by changing the model, but by connecting it to the outside world: • retrieving fresh data • grounding answers in facts • using tools and memory to stay accurate The goal is not to overload a prompt. It is to make the model smarter about what stays active and what gets offloaded. This is what separates basic LLM Q and A from real production systems. To do this right, you need six components working together 👇 ⸻ 1. Agents 🤖 The decision makers. Agents evaluate what they know, decide what they need, choose the right tools, and recover when things go wrong. ⸻ 2. Query Augmentation 🔎 Turning messy user input into precise intent. If the system does not know exactly what the user is asking, everything downstream fails. ⸻ 3. Retrieval 📚 The bridge from the model to your real data. This is chunking, indexing, and fetching the right facts with the right balance of precision and context. ⸻ 4. Prompting Techniques 🧭 Guiding the model with clear reasoning instructions. Chain of Thought, Few shot examples, ReAct style prompting, and more. ⸻ 5. Memory 🧠 Short term and long term. Your app needs to remember past interactions and keep persistent knowledge available when needed. ⸻ 6. Tools 🔧 The action layer. APIs, code execution, web browsing, database calls. This is how your system moves from answering questions to actually performing work. ⸻ This is far more advanced than classic RAG. This is how production systems maintain coherence, access live data, reduce hallucinations, and actually get work done. If you want more breakdowns like this on LLM architecture, RAG systems, and AI engineering, follow my profile here on LinkedIn.

Just finished reading an amazing book: AI Engineering by Chip Huyen. Here’s the quickest (and most agile) way to build LLM products: 1. Define your product goals Pick a small, very clear problem to solve (unless you're building a general chatbot). Identify use case and business objectives. Clarify user needs and domain requirements. 2. Select the foundation model Don’t waste time training your own at the start. Evaluate models for domain relevance, task capability, cost, and privacy. Decide on open source vs. proprietary options. 3. Gather and filter data Collect high-quality, relevant data. Remove bias, toxic content, and irrelevant domains. 4. Evaluate baseline model performance Use key metrics: cross-entropy, perplexity, accuracy, semantic similarity. Set up evaluation benchmarks and rubrics. 5. Adapt the model for your task Start with prompt engineering (quick, cost-effective, doesn’t change model weights): craft detailed instructions, provide examples, and specify output formats. Use RAG if your application needs strong grounding and frequently updated factual data: integrate external data sources for richer context. Prompt-tuning isn’t a bad idea either. Still getting hallucinations? Try “abstention”—having the model say “I don’t know” instead of guessing. 6. Fine-tune (only if you have a strong case for it) Train on domain/task-specific data for better performance. Use model distillation for cost-efficient deployment. 7. Implement safety and robustness Protect against prompt injection, jailbreaks, and extraction attacks. Add safety guardrails and monitor for security risks. 8. Build memory and context systems Design short-term and long-term memory (context windows, external databases). Enable continuity across user sessions. 9. Monitor and maintain Continuously track model performance, drift, evaluation metrics, business impact, token usage, etc. Update the model, prompts, and data based on user feedback and changing requirements. Observability is key! 10. Test, Test, Test! Use LLM judges, human-in-the-loop strategies; iterate in small cycles. A/B test in small iterations: see what breaks, patch, and move on. A simple GUI or CLI wrapper is just fine for your MVP. Keep scope under control—LLM products can be tempting to expand, but restraint is crucial! Fastest way: Build an LLM optimized for a single use case first. Once that works, adding new use cases becomes much easier. https://lnkd.in/ghuHNP7t Summary video here -> https://lnkd.in/g6fPsqUR Chip Huyen, #AiEngineering #LLM #GenAI #Oreilly #ContinuousLEarning #ProductManagersinAI

Free 𝗧𝗲𝗺𝗽𝗹𝗮𝘁𝗲 𝗳𝗼𝗿 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮𝗻𝗱 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 - steal it with pride! I have been developing Agentic Systems for around two years now. The same patterns keep emerging. Today, I am sharing my system of how to approach development of LLM based applications from idea to production. Let’s zoom in: 𝟭. Define a problem you want to solve: is GenAI even needed? 𝟮. Build a Prototype: figure out if the solution is feasible. 𝟯. Define Performance Metrics: you must have output metrics defined for how you will measure success of your application. 𝟰. Define Evals: split the above into smaller input metrics that can move the key metrics forward. Decompose them into tasks that could be automated and move the given input metrics. Define Evals for each. Store the Evals in your Observability Platform. ℹ️ Steps 𝟭. - 𝟰. are where AI Product Managers can help, but can also be handled by AI Engineers. 𝟱. Build a PoC: it can be simple (excel sheet) or more complex (user facing UI). Regardless of what it is, expose it to the users for feedback as soon as possible. 𝟲. Instrument your application: gather traces and human feedback and store it in an Observability Platform next to previously stored Evals. 𝟳. Run Evals on traced data: traces contain inputs and outputs of your application, run evals on top of them. 𝟴. Analyse Failing Evals and negative user feedback: this data is gold as it specifically pinpoints where the Agentic System needs improvement. 𝟵. Use data from the previous step to improve your application - prompt engineer, improve AI system topology, finetune models etc. Make sure that the changes move Evals into the right direction. 𝟭𝟬. Build and expose the improved application to the users. 𝟭𝟭. Monitor the application in production: this comes out of the box - you have implemented evaluations and traces for development purposes, they can be reused for monitoring. Configure specific alerting thresholds and enjoy the peace of mind. ✅ 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗼𝗳 𝘆𝗼𝘂𝗿 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻: ➡️ Run steps 𝟲. - 𝟭𝟬. to continuously improve and evolve your application. ➡️ As you build up in complexity, new requirements can be added to the same application, this includes running steps 𝟭. - 𝟱. and attaching the new logic as routes to your Agentic System. ➡️ You start off with a simple Chatbot and add a route that can classify user queries to take action (e.g. add items to a shopping cart). I will be teaching how to apply this system hands-on and in detail as part of End-to-End AI Engineering Bootcamp (𝟭𝟬% 𝗱𝗶𝘀𝗰𝗼𝘂𝗻𝘁 𝗰𝗼𝗱𝗲: Kickoff10 ): https://lnkd.in/dGVhxAD9 What is your experience in evolving Agentic Systems? Let me know in the comments 👇 #LLM #AI #MachineLearning