How LLM Accuracy Shapes Software Development

LLMs are great for data processing, but using new techniques doesn't mean you get to abandon old best practices. The precision and accuracy of LLMs still need to be monitored and maintained, just like with any other AI model. Tips for maintaining accuracy and precision with LLMs: • Define within your team EXACTLY what the desired output looks like. Any area of ambiguity should be resolved with a concrete answer. Even if the business "doesn't care," you should define a behavior. Letting the LLM make these decisions for you leads to high variance/low precision models that are difficult to monitor. • Understand that the most gorgeously-written, seemingly clear and concise prompts can still produce trash. LLMs are not people and don't follow directions like people do. You have to test your prompts over and over and over, no matter how good they look. • Make small prompt changes and carefully monitor each change. Changes should be version tracked and vetted by other developers. • A small change in one part of the prompt can cause seemingly-unrelated regressions (again, LLMs are not people). Regression tests are essential for EVERY change. Organize a list of test case inputs, including those that demonstrate previously-fixed bugs and test your prompt against them. • Test cases should include "controls" where the prompt has historically performed well. Any change to the control output should be studied and any incorrect change is a test failure. • Regression tests should have a single documented bug and clearly-defined success/failure metrics. "If the output contains A, then pass. If output contains B, then fail." This makes it easy to quickly mark regression tests as pass/fail (ideally, automating this process). If a different failure/bug is noted, then it should still be fixed, but separately, and pulled out into a separate test. Any other tips for working with LLMs and data processing?

We know LLMs can substantially improve developer productivity. But the outcomes are not consistent. An extensive research review uncovers specific lessons on how best to use LLMs to amplify developer outcomes. 💡 Leverage LLMs for Improved Productivity. LLMs enable programmers to accomplish tasks faster, with studies reporting up to a 30% reduction in task completion times for routine coding activities. In one study, users completed 20% more tasks using LLM assistance compared to manual coding alone. However, these gains vary based on task complexity and user expertise; for complex tasks, time spent understanding LLM responses can offset productivity improvements. Tailored training can help users maximize these advantages. 🧠 Encourage Prompt Experimentation for Better Outputs. LLMs respond variably to phrasing and context, with studies showing that elaborated prompts led to 50% higher response accuracy compared to single-shot queries. For instance, users who refined prompts by breaking tasks into subtasks achieved superior outputs in 68% of cases. Organizations can build libraries of optimized prompts to standardize and enhance LLM usage across teams. 🔍 Balance LLM Use with Manual Effort. A hybrid approach—blending LLM responses with manual coding—was shown to improve solution quality in 75% of observed cases. For example, users often relied on LLMs to handle repetitive debugging tasks while manually reviewing complex algorithmic code. This strategy not only reduces cognitive load but also helps maintain the accuracy and reliability of final outputs. 📊 Tailor Metrics to Evaluate Human-AI Synergy. Metrics such as task completion rates, error counts, and code review times reveal the tangible impacts of LLMs. Studies found that LLM-assisted teams completed 25% more projects with 40% fewer errors compared to traditional methods. Pre- and post-test evaluations of users' learning showed a 30% improvement in conceptual understanding when LLMs were used effectively, highlighting the need for consistent performance benchmarking. 🚧 Mitigate Risks in LLM Use for Security. LLMs can inadvertently generate insecure code, with 20% of outputs in one study containing vulnerabilities like unchecked user inputs. However, when paired with automated code review tools, error rates dropped by 35%. To reduce risks, developers should combine LLMs with rigorous testing protocols and ensure their prompts explicitly address security considerations. 💡 Rethink Learning with LLMs. While LLMs improved learning outcomes in tasks requiring code comprehension by 32%, they sometimes hindered manual coding skill development, as seen in studies where post-LLM groups performed worse in syntax-based assessments. Educators can mitigate this by integrating LLMs into assignments that focus on problem-solving while requiring manual coding for foundational skills, ensuring balanced learning trajectories. Link to paper in comments.

🧠 If you're building apps with LLMs, this paper is a must-read. Researchers at Microsoft and Salesforce recently released LLMs Get Lost in Multi-Turn Conversation — and the findings resonate with our experience at Vellum. They ran 200,000+ simulations across 15 top models, comparing performance on the same task in two modes: - Single-turn (user provides a well-specified prompt upfront) - Multi-turn (user reveals task requirements gradually — like real users do) The result? ✅ 90% avg accuracy in single-turn 💬 65% avg accuracy in multi-turn 🔻 -39% performance drop across the board 😬 Unreliability more than doubled Even the best models get lost when the task unfolds over multiple messages. They latch onto early assumptions, generate bloated answers, and fail to adapt when more info arrives. For application builders, this changes how we think about evaluation and reliability: - One-shot prompt benchmarks ≠ user reality - Multi-turn behavior needs to be a first-class test case - Agents and wrappers won’t fix everything — the underlying model still gets confused This paper validates something we've seen in the wild: the moment users interact conversationally, reliability tanks — unless you're deliberate about managing context, fallback strategies, and prompt structure. 📌 If you’re building on LLMs, read this. Test differently. Optimize for the real-world path, not the happy path.

Please stop building multi-agent systems. Autonomy means nothing if the system can’t repeat its own success 𝟭. 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗻𝗲𝗲𝗱𝘀 "𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴" It isn’t about “can it solve the problem?” It’s “can it solve the problem under real constraints, and still make business sense?” ↳ Constraints: cost, latency, accuracy, compliance, security, privacy, ethics ↳ Value: measurable user impact (time saved, risk reduced, revenue unlocked) ↳ Unit economics: margins today or a credible path soon Add even one constraint, and the search space explodes. Add scale, and it gets harder again. 𝟮. 𝗟𝗟𝗠𝘀 𝗮𝗿𝗲 𝗲𝘅𝗰𝗲𝗹𝗹𝗲𝗻𝘁 𝗮𝘁 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲, 𝘀𝗵𝗮𝗸𝘆 𝗮𝘁 𝗮𝗱𝗵𝗲𝗿𝗲𝗻𝗰𝗲 The creative variability we love trades off with reliability. ↳ Non-deterministic outputs ↳ Instruction drift across long tasks ↳ Sensitivity to prompt/context formatting Great for ideation and synthesis; fragile for strict, long-horizon execution. 𝟯. 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲-𝗴𝗿𝗮𝗱𝗲 𝗺𝗲𝗮𝗻𝘀 𝗼𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 To tame non-determinism, you have to add structure. A lot of it. ↳ Task decomposition and state: break work into verifiable steps, persist state ↳ Data layer: sourcing → cleaning → chunking → embeddings → indexing (RAG) ↳ Prompt lifecycle: versioning, testing, registries, rollout/rollback ↳ Model routing & caching: pick the smallest model that meets quality, reuse context ↳ Evals & observability: ground-truth tests, regression suites, traces, guardrails ↳ The triangle you must balance every day: accuracy ↔ cost ↔ latency Yes, the “mammoth thinking model” can brute-force quality, only if your users can wait and you can eat the bill. Most can’t. 𝟰. 𝗧𝗿𝗲𝗮𝘁 𝗔𝗜 𝗮𝘀 𝗮 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁 𝗶𝗻 𝗮 𝘀𝘆𝘀𝘁𝗲𝗺, 𝘁𝗵𝗲𝗻 𝗰𝗵𝗼𝗼𝘀𝗲 𝘁𝗵𝗲 𝘀𝗶𝗺𝗽𝗹𝗲𝘀𝘁 𝘁𝗵𝗶𝗻𝗴 𝘁𝗵𝗮𝘁 𝘄𝗼𝗿𝗸𝘀 For most production use cases: ↳ 𝗥𝗔𝗚 𝘄𝗶𝘁𝗵 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀 > 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 (Tight retrieval, reranking, and schema constraints beat free-roaming planners.) ↳ 𝗛𝗲𝘂𝗿𝗶𝘀𝘁𝗶𝗰/𝗺𝗲𝘁𝗿𝗶𝗰-𝗯𝗮𝘀𝗲𝗱 𝗲𝘃𝗮𝗹𝘀 𝘄𝗶𝘁𝗵 𝗵𝗶𝗴𝗵-𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗴𝗿𝗼𝘂𝗻𝗱 𝘁𝗿𝘂𝘁𝗵 > 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗷𝘂𝗱𝗴𝗲 (Use the model to propose, not police, unless you’ve calibrated it carefully.) ↳ 𝗗𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 𝗮𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗟𝗟𝗠 𝗮𝘁 𝘁𝗵𝗲 𝘀𝗲𝗮𝗺𝘀 > 𝗠𝘂𝗹𝘁𝗶-𝗮𝗴𝗲𝗻𝘁 𝗲𝘃𝗲𝗿𝘆𝘁𝗵𝗶𝗻𝗴 (Let the LLM read/plan/rewrite; let code and tools execute.) ↳ 𝗖𝗹𝗮𝘀𝘀𝗶𝗰 𝗠𝗟 𝗼𝗿 𝗿𝘂𝗹𝗲𝘀 𝗳𝗼𝗿 𝘀𝘁𝗮𝗯𝗹𝗲 𝘀𝗶𝗴𝗻𝗮𝗹𝘀 > 𝗠𝗮𝗻𝗮𝗴𝗶𝗻𝗴 𝗟𝗟𝗠 𝘀𝘁𝗼𝗰𝗵𝗮𝘀𝘁𝗶𝗰 𝗵𝗲𝗹𝗹 (Don’t use a bazooka to swat a fly; it's harder to aim) LLMs are powerful, but they’re one part of a disciplined software system. Engineer the system first. Insert the model where it actually improves reliability, speed, cost or efficiency. ♻️ Repost to share these insights.

You're under pressure to deliver on AI's promise while navigating vendor hype and technical limitations. Your leadership team wants ROI, your employees want tools that work, and you're desperately trying to separate AI reality from market fiction. And now, you're learning the news that the AI foundation everyone's building on was never solid, and research shows it's actively getting worse. Wait... what? Doesn't emerging technology typically improve over time? It's called "model collapse". We've all heard "garbage in, garbage out." This is the compounding of that. LLMs trained on their own outputs gradually lose accuracy, diversity, and reliability. Errors compound across successive model generations. A Nature 2024 paper describes this as models becoming "poisoned with their own projection of reality." But here's the truth. LLMs were always questionable for business decisions. They were trained on random internet content. Would you base quarterly projections on Wikipedia articles? Model collapse just compounds this fundamental problem. What does this mean for your AI strategy, since much it is likely based on the use of LLMs? It comes down to the decisions you make at the beginning. Most of us are rushing to launch the latest model, when we should be looking at what's best for the use case at hand. First things first, deploy LLMs when you can afford to be wrong: ✔️ Brainstorming and ideation ✔️ First-draft content (with human editing) ✔️ Low-stakes support services Stop using LLMs when being wrong carries costs: 🛑 Financial analysis and reporting 🛑 Legal compliance 🛑 Safety-critical procedures I'm not saying LLMs are useless. Agentic AI will be driven by them, but there are significant achievements in small language models (SMLs) and other foundational, open-source models that perform just as well, even better, at particular tasks. So here's what you need to do as part of your AI strategy: 1️⃣ Classify your AI use cases: For all use cases, classify by accuracy required. You can still use LLMs, but that just means you need more validation around outputs 2️⃣ Assess LLM vs. SML strategy: Evaluate smaller, domain-specific language models for critical functions and experiment with them against LLMS and see how they perform 3️⃣ Consider deterministic alternatives: For calculations, and workflows requiring consistency, rule-based solution or deterministic AI solutions may be better 4️⃣ Design hybrid architectures: Combine specialized models with deterministic fallbacks. This area is moving fast; flexibility is key The bottom line? Your success will be measured not by how quickly you adopt every AI tool, but by how strategically you deploy AI where it creates value and reliability. Model Collapse Research: https://lnkd.in/gUTChswk Signs of Model Collapse: https://lnkd.in/g5ZpAk89 #ai #innovation #future                                

Achieving 3x-25x Performance Gains for High-Quality, AI-Powered Data Analysis Asking complex data questions in plain English and getting precise answers feels like magic, but it’s technically challenging. One of my jobs is analyzing the health of numerous programs. To make that easier we are building an AI app with Sapient Slingshot that answers natural language queries by generating and executing code on project/program health data. The challenge is that this process needs to be both fast and reliable. We started with gemini-2.5-pro, but 50+ second response times and inconsistent results made it unsuitable for interactive use. Our goal: reduce latency without sacrificing accuracy. The New Bottleneck: Tuning "Think Time" Traditional optimization targets code execution, but in AI apps, the real bottleneck is LLM "think time", i.e. the delay in generating correct code on the fly. Here are some techniques we used to cut think time while maintaining output quality: ① Context-Rich Prompts Accuracy starts with context. We dynamically create prompts for each query: ➜ Pre-Processing Logic: We pre-generate any code that doesn't need "intelligence" so that LLM doesn't have to ➜ Dynamic Data-Awareness: Prompts include full schema, sample data, and value stats to give the model a full view. ➜ Domain Templates: We tailor prompts for specific ontology like "Client satisfaction" or "Cycle Time" or "Quality". This reduces errors and latency, improving codegen quality from the first try. ② Structured Code Generation Even with great context, LLMs can output messy code. We guide query structure explicitly: ➜ Simple queries: Direct the LLM to generate a single line chained pandas expression. ➜ Complex queries : Direct the LLM to generate two lines, one for processing, one for the final result Clear patterns ensure clean, reliable output. ③ Two-Tiered Caching for Speed Once accuracy was reliable, we tackled speed with intelligent caching: ➜ Tier 1: Helper Cache – 3x Faster ⊙ Find a semantically similar past query ⊙ Use a faster model (e.g. gemini-2.5-flash) ⊙ Include the past query and code as a one-shot prompt This cut response times from 50+s to <15s while maintaining accuracy. ➜ Tier 2: Lightning Cache – 25x Faster ⊙ Detect duplicates for exact or near matches ⊙ Reuse validated code ⊙ Execute instantly, skipping the LLM This brought response times to ~2 seconds for repeated queries. ④ Advanced Memory Architecture ➜ Graph Memory (Neo4j via Graphiti): Stores query history, code, and relationships for fast, structured retrieval. ➜ High-Quality Embeddings: We use BAAI/bge-large-en-v1.5 to match queries by true meaning. ➜ Conversational Context: Full session history is stored, so prompts reflect recent interactions, enabling seamless follow-ups. By combining rich context, structured code, caching, and smart memory, we can build AI systems that deliver natural language querying with the speed and reliability that we, as users, expect of it.

92% of U.S. developers now use LLMs like ChatGPT for daily coding tasks. But new research reveals a concerning blind spot: these tools miss 60% of security vulnerabilities, even in the best case scenarios. The problem is subtle but serious: When developers share code with LLMs for debugging or optimization, the AI often provides functional solutions while completely ignoring security flaws. This creates a false sense of security—code works as expected, but remains vulnerable to attacks. Key findings from the research study include: ✨ GPT-4 (the best performer at the time) only warned about security issues 40% of the time ✨ Detection rates dropped to just 16.6% for novel code patterns ✨ Performance heavily depended on whether similar code appeared in training data For engineering teams, these have serious implications on the health and stability of your product. This means teams need to adapt in a few ways: 1️⃣ Explicitly request security considerations in LLM prompts 2️⃣ Integrate static analysis tools into AI-assisted workflows 3️⃣ Update code review processes to account for LLM blind spots The takeaway: LLMs are powerful development tools, but they're not security tools. As we embrace AI assistance, we need security processes that work alongside—not instead of—human expertise.

Current benchmark scores for LLMs in software engineering are inflated by memorization rather than genuine coding ability. Models achieve high accuracy on tasks like bug localization and patch generation not by reasoning over code and issue descriptions, but by recalling specific issue-file pairs or reproducing memorized code patterns. Controlled experiments across multiple benchmarks reveal two types of memorization: instance-specific, where exact examples are memorized, and repository-bias, where uneven familiarity with certain codebases skews performance. Even when stripped of contextual clues, models perform significantly better on curated datasets like SWE-Bench-Verified than on new, unexposed tasks, confirming overfitting to benchmark-specific data. Metrics such as filtered accuracy and 5-gram similarity reinforce that performance often reflects exposure, not transferable problem-solving skill. A differential testing approach highlights the urgent need for contamination-resistant benchmarks to more accurately assess true software engineering competence. https://lnkd.in/gyJCpM9g

When Probabilities Compound: Why Agent Accuracy Breaks Down The obvious thing about LLMs I still think isn't talked about enough. In traditional software, you can run the same input a million times and get the exact same output. That’s determinism. CPUs are the archetype here—perfectly predictable, clockwork precise. LLMs don’t work that way. They’re probabilistic. Every output is a weighted guess over possible tokens. You can tune the randomness (temperature), but even at zero, small differences in context or prompt can shift results. GPUs—built for parallel matrix multiplications—are what make this possible at scale, but they’re also part of the probabilistic paradigm that’s replacing deterministic computation in many workflows. Many people I talk to every day in AI still haven’t wrapped their heads around this enough. As an Industrial Engineer by degree, the statistics hits you in the face. Now add agents into the mix. Those deep in AI know this intimately but newer founders and builders in the agentic space are learning this the hard way.  One LLM call → slight uncertainty. Chain 5–10 LLM calls across an agent workflow → you’re compounding that uncertainty. It’s like multiplying probabilities less than 1 together—the overall accuracy drops fast. You have errors compounding. This matters if you’re building with multi-step reasoning, tool use, or autonomous agents: Your workflow is only as reliable as the weakest probabilistic link Guardrails, verification, and redundancy aren’t “nice-to-haves”—they’re architecture The longer your chain of calls, the more you need to design for failure modes. Probabilistic systems open up new possibilities that deterministic systems never could. But if you don’t understand how probabilities compound, you’ll overestimate what’s possible—and ship something brittle. To me, this is what squares the disconnect I’m hearing in market where in many ways we are “ahead” of where we thought might be with agents and in many ways we are “behind.” As VCs, we’re watching the founders who design for this reality, not against it. They’re the ones building AI systems that will stand up in production. For entertainment value and a reminder, three screenshots below, courtesy of a friend all wrong but presented by Google Gemini as the answer to a simple question. Some you can see in plain sight they are wrong but some you have to know the correct answer (tallest building one, which is WAY off, to know). We still aren't that accurate on a single LLM call, let alone a daisy chain of agents. 💭 Curious: How are you mitigating compounded uncertainty in your LLM workflows? What deterministic tools are you adding in to improve accuracy?

In modern software development, we don't just guess if our code works. We write unit tests, run integration tests, and build CI/CD pipelines. We replaced manual guesswork with rigorous, automated validation. So why are many of us still in the "guesswork" phase with LLM prompts? The common workflow is a manual loop : tweak a prompt, test it, eyeball the result, and tweak it again. This is artisanal, slow, and doesn't scale. A prompt that works today might break tomorrow with a slight model update. It’s not an engineering discipline. The paradigm shift we need is Systematic Prompt Optimization. This is the move from "prompt art" to "prompt science." It’s about treating a prompt not as a magic incantation, but as a key component of a system that can be algorithmically tested, measured, and improved. The framework for this is surprisingly simple and powerful: 1./ Hypothesis (Your Base Prompt): Your initial, best-guess prompt. 2./ Ground Truth (An Evaluation Dataset): A set of inputs and ideal outputs that define success for your use case. 3./ Objective Function (An Evaluator): A measurable score for success (e.g., accuracy, semantic similarity, factuality). 4./ Optimizer: An algorithm that intelligently searches the vast space of possible prompt variations to find the one that maximizes your objective function. This approach is a repeatable, data-driven process. It allows you to prove why one prompt is better than another and ensures your system is robust. I've been exploring frameworks that enable this, and Comet's Opik is a fascinating, concrete example of this principle in action. It provides the optimizer and structure to automate this entire loop. Check here: https://lnkd.in/dZEfCW6S By adopting this mindset, we're not just writing better prompts. We're building more reliable, maintainable, and predictable AI systems. What steps is your team taking to bring more engineering discipline to your work with LLMs? #llm #ai #optimization #agents