Improving LLM Coding Accuracy with Code Intelligence

We improved Cline, a popular open-source coding agent, by +15% accuracy on SWE-Bench —  without retraining the LLM, changing any tools, or modifying the architecture whatsoever. How? All we did was optimize its ruleset, in ./clinerules — a user defined section for developers to add custom instructions to the system prompt, just like .cursor/rules in Cursor or CLAUDE.md in Claude Code. Using our algorithm, Prompt Learning, we automatically refined these rules across a feedback loop powered by GPT-5. What is Prompt Learning? It’s an optimization algorithm that improves prompts, not models. Inspired by RL, it follows an action → evaluation → improvement loop — but instead of gradients, it uses Meta Prompting: feeding a prompt into an LLM and asking it to make it better. We add a key twist — LLM-generated feedback explaining why outputs were right or wrong, giving the optimizer richer signal to refine future prompts. The result: measurable gains in accuracy, zero retraining. You can use it in Arize AX or the Prompt Learning SDK. Here’s how we brought GPT-4.1’s performance on SWE-Bench Lite to near state-of-the-art levels — matching Claude Sonnet 4-5 — purely through ruleset optimization. Last time, we optimized Plan Mode - but this time, we optimized over Act Mode - giving Cline full permissions to read, write, and edit code files, and testing its accuracy on SWE Bench Lite. Our optimization loop: 1️⃣ Run Cline on SWE-Bench Lite (150 train, 150 test) and record its train/test accuracy. 2️⃣ Collect the patches it produces and verify correctness via unit tests. 3️⃣ Use GPT-5 to explain why each fix succeeded or failed on the training set. 4️⃣ Feed those training evals — along with Cline’s system prompt and current ruleset — into a Meta-Prompt LLM to generate an improved ruleset.  5️⃣ Update ./clinerules, re-run, and repeat. The results: Sonnet 4-5 saw a modest +6% training and +0.7% test gain — already near saturation — while GPT-4.1 improved 14–15% in both, reaching near-Sonnet performance (34% vs 36%) through ruleset optimization alone in just two loops! These results highlight how prompt optimization alone can deliver system-level gains — no retraining, no new tools, no architecture changes. In just two optimization loops, Prompt Learning closed much of the gap between GPT-4.1 and Sonnet-level performance, proving how fast and data-efficient instruction-level optimization can be. And of course, we used Arize Phoenix to run LLM evals on Cline’s code and track experiments across optimization runs. Code here: https://lnkd.in/eDejFy6N

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]

Meta just dropped a new kind of code model; and it's not just bigger. It's different. The new Code World Model (CWM), a 32B parameter LLM for code generation is not "just another code model." What makes it different? CWM was trained not only on code, but on what code does at runtime. Most LLMs learn code like they learn prose: predict the next token. CWM learns code like developers do; by simulating its execution. This shift is critical because: - When humans debug or write code, we think in terms of state changes, side effects, and what happens next. - CWM learns from execution traces of Python functions and agentic behaviors in Dockerized Bash environments. It doesn’t just guess the next line; it reasons like it’s living inside the terminal. This unlocks: - Stronger reasoning in multi-step problems - Simulation-based debugging ; More accurate code generation in real-world workflows - Potential for autonomous “neural debuggers” that think in traces, not just tokens On benchmarks, it’s already competitive: - 68.6% on LiveCodeBench v5 - 76% on AIME 2024 - 65.8% on SWE-bench Verified And it's open weights. Meta is betting that world modeling + RL fine-tuning is the next frontier for coding LLMs; not just scale. Is this a glimpse of what post-token-prediction AI looks like? Get started with the links below: - Tech Report: https://lnkd.in/eV7YirjC - Model Weights: https://lnkd.in/e2CTzsxr - On Huggingface: https://lnkd.in/e_S4R-P4 - Inference Code: https://lnkd.in/eVHeW8VV ___ If you like this content and it resonates, follow me Armand Ruiz for more like it.

I’ve been working on a massive prompt that extracts structured data from unstructured text. It's effectively a program, developed over the course of weeks, in plain English. Each instruction is precise. The output format is strict. The logic flows. It should Just Work™. And the model? Ignores large swaths of it. Not randomly, but consistently and stubbornly. This isn't a "program," it's a probability engine with auto-complete. This is because LLMs don’t "read" like we do, or execute prompts like a program does. They run everything through the "attention mechanism," which mathematically weighs which tokens matter in relation to others. Technically speaking: Each token is transformed into a query, key, and value vector. The model calculates dot products between the query vector and all key vectors to assign weights. Basically: "How relevant is this other token to what I’m doing right now?" Then it averages the values using those weights and moves on. No state. No memory. Just a rolling calculation over a sliding window of opaquely-chosen context. It's kind of tragic, honestly. You build this beautifully precise setup, but because your detailed instructions are buried in the middle of a long prompt -- or phrased too much like background noise -- they get low scores. The model literally pays less attention to them. We thought we were vibe coding, but the real vibe coder was the LLM all along! So how to fix it? Don’t just write accurate instructions. Write ATTENTION-WORTHY ones. - 🔁 Repeat key patterns. Repetition increases token relevance, especially when you're relying on specific phrasing to guide the model's output. - 🔝 Push constraints to the top. Instructions buried deep in the prompt get lower attention scores. Front-load critical rules so they have a better chance of sticking. - 🗂️ Use structure to force salience. Consistent headers, delimiters, and formatting cues help key sections stand out. Markdown, line breaks, and even ALL CAPS (sparingly) can help direct the model's focus to what actually matters. - ✂️ Cut irrelevant context. The less junk in the prompt, the more likely your real instructions are to be noticed and followed. You're not teaching a model. You're gaming a scoring function.

91.3% accuracy vs 0%. Same model. Same task. The only difference: treating your prompt as code instead of text. Recursive Language Models (RLMs) from MIT have completely changed how I think about handling long context in LLMs. Instead of cramming everything into the context window, RLMs treat your prompt as part of the 𝘦𝘯𝘷𝘪𝘳𝘰𝘯𝘮𝘦𝘯𝘵 that the model can programmatically explore. 𝗧𝗵𝗲 𝗖𝗼𝗿𝗲 𝗜𝗻𝘀𝗶𝗴𝗵𝘁 Once you hit the context limit in an LLM, you're done. But LLMs are trained for code as well, right? Why not use their coding skills for more than just coding? 1. Load your prompt as a 𝘷𝘢𝘳𝘪𝘢𝘣𝘭𝘦 in a REPL programming environment 2. Give the model tools to peek into, decompose, and recursively process parts of that variable 3. Let the model write 𝘤𝘰𝘥𝘦 that calls itself on programmatic slices of the input This enables the model to handle prompts that are literally 100x longer than its context window. The 𝗿𝗲𝗰𝘂𝗿𝘀𝗶𝘃𝗲 element is the key insight here - the LLM can call itself (or a smaller subagent) for smaller tasks, allowing it to batch and concatenate results to answer complex questions. 𝗘𝘅𝗮𝗺𝗽𝗹𝗲 I tested it in Python (via DSPy), input the full alice in wonderland book and asked it to give a sentiment analysis of the openings of each chapter. The LLM: 1. Explored the prompt (book) to see how the chapter headings were formatted 2. Implemented regex to split the full string into chunks before/after each chapter heading 3. Invoked the LLM sub-agent on each paragraph to analyse the sentiment Even if the full prompt can't fit into history, LLMs have notoriously suffered from context rot. This approach enabled each task to be separately analysed by the sub-agent, each having no knowledge of the greater task. 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 • RLMs successfully process inputs up to 𝘁𝘄𝗼 𝗼𝗿𝗱𝗲𝗿𝘀 𝗼𝗳 𝗺𝗮𝗴𝗻𝗶𝘁𝘂𝗱𝗲 beyond model context windows • On BrowseComp-Plus (6-11M tokens), RLM(GPT-5) achieved 91.3% accuracy vs 0% for the base model RLMs aren't perfect. The inference cost has high variance - median costs are comparable to base models, but some trajectories explode to 3x+ the cost due to long recursive chains. I also found, as the authors note in the appendix, that the models continue analysing well past when they had already found an answer. My hunch is that each LLM invocation always wants to do 𝘴𝘰𝘮𝘦𝘵𝘩𝘪𝘯𝘨, even if that something has already been done. It always wants to check its answer. Because of how they're trained, LLMs never just say "Okay, done!". The paper demonstrates that with better training (especially on-policy rollouts at scale), native RLMs could become far more efficient than current implementations suggest. I'll be extremely excited if this becomes a core part of model training, building custom models that excel at managing their prompt with code. Read the paper: https://lnkd.in/eq_xUJvJ

Code generation poses distinct challenges compared to common Natural Language tasks (NLP).  Conventional prompt engineering techniques, while effective in NLP, exhibit limited efficacy within the intricate domain of code synthesis. This is one reason why we continuously see code-specific LLM-oriented innovation. Specifically, LLMs demonstrated shortcomings when tackling coding problems from competitions such as SWE-bench and Code-Contest using naive prompting such as single-prompt or chain-of-thought methodologies, frequently producing erroneous or insufficiently generic code. To address these limitations, at CodiumAI, we introduced AlphaCodium, a novel test-driven, iterative framework designed to enhance the performance of LLM-based algorithms in code generation. Evaluated on the challenging Code-Contests benchmark, AlphaCodium consistently outperforms advanced (yet straightforward) prompting using state-of-the-art models, including GPT-4, and even Gemini AlphaCode 2 while demanding fewer computational resources and without fine-tuning.  For instance, #AlphaCodium elevated GPT-4's accuracy from 19% to 44% on the validation set. AlphaCodium is an open-source project that works with most leading models. Interestingly, the accuracy gaps presented by leading models change and commonly shrink when using flow-engineering instead of prompt-engineering only. We will keep pushing the boundaries of intelligent software development, and using #benchmarks is a great way to achieve and demonstrate progress. Which benchmark best represents your real-world #coding and software development challenges?

Tired of your LLM just repeating the same mistakes when retries fail? Simple retry strategies often just multiply costs without improving reliability when models fail in consistent ways. You've built validation for structured LLM outputs, but when validation fails and you retry the exact same prompt, you're essentially asking the model to guess differently. Without feedback about what went wrong, you're wasting compute and adding latency while hoping for random success. A smarter approach feeds errors back to the model, creating a self-correcting loop. Effective AI Engineering #13: Error Reinsertion for Smarter LLM Retries 👇 The Problem ❌ Many developers implement basic retry mechanisms that blindly repeat the same prompt after a failure: [Code example - see attached image] Why this approach falls short: - Wasteful Compute: Repeatedly sending the same prompt when validation fails just multiplies costs without improving chances of success. - Same Mistakes: LLMs tend to be consistent - if they misunderstand your requirements the first time, they'll likely make the same errors on retry. - Longer Latency: Users wait through multiple failed attempts with no adaptation strategy.Beyond Blind Repetition: Making Your LLM Retries Smarter with Error Feedback. - No Learning Loop: The model never receives feedback about what went wrong, missing the opportunity to improve. The Solution: Error Reinsertion for Adaptive Retries ✅ A better approach is to reinsert error information into subsequent retry attempts, giving the model context to improve its response: [Code example - see attached image] Why this approach works better: - Adaptive Learning: The model receives feedback about specific validation failures, allowing it to correct its mistakes. - Higher Success Rate: By feeding error context back to the model, retry attempts become increasingly likely to succeed. - Resource Efficiency: Instead of hoping for random variation, each retry has a higher probability of success, reducing overall attempt count. - Improved User Experience: Faster resolution of errors means less waiting for valid responses. The Takeaway Stop treating LLM retries as mere repetition and implement error reinsertion to create a feedback loop. By telling the model exactly what went wrong, you create a self-correcting system that improves with each attempt. This approach makes your AI applications more reliable while reducing unnecessary compute and latency.

How to actually code with LLMs in 2026. Not the hype. What's working for engineers who ship: 1️⃣ 𝗦𝗽𝗲𝗰 𝗯𝗲𝗳𝗼𝗿𝗲 𝗰𝗼𝗱𝗲 Don't throw wishes at the LLM. → Describe the idea → Let the AI ask questions until requirements are clear → Compile into spec.md → Generate step-by-step plan → Then code It's "waterfall in 15 minutes." 2️⃣ 𝗦𝗺𝗮𝗹𝗹 𝗰𝗵𝘂𝗻𝗸𝘀 Ask for too much = jumbled mess. "Like 10 devs worked on it without talking." One function. One bug. One feature. Then next. 3️⃣ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗽𝗮𝗰𝗸𝗶𝗻𝗴 LLMs are only as good as what you show them. → Relevant code → API docs → Known pitfalls → Preferred approaches Don't make the AI guess. 4️⃣ 𝗠𝗼𝗱𝗲𝗹 𝗺𝘂𝘀𝗶𝗰𝗮𝗹 𝗰𝗵𝗮𝗶𝗿𝘀 Each model has blind spots. Stuck? Copy the same prompt to another model. Sometimes a second opinion is all you need. 5️⃣ 𝗛𝘂𝗺𝗮𝗻 𝗶𝗻 𝘁𝗵𝗲 𝗹𝗼𝗼𝗽 AI writes with complete conviction. Including bugs. Including nonsense. Treat every snippet like junior dev code. Read it. Run it. Test it. 6️⃣ 𝗖𝗼𝗺𝗺𝗶𝘁 𝗹𝗶𝗸𝗲 𝘀𝗮𝘃𝗲 𝗽𝗼𝗶𝗻𝘁𝘀 AI generates fast. Veers off course fast. Commit after each small task. Your safety net when AI goes sideways. 7️⃣ 𝗥𝘂𝗹𝗲𝘀 𝗳𝗶𝗹𝗲𝘀 Use CLAUDE.md, GEMINI.md, or .cursorrules. → Your coding standards → Your patterns → Your constraints Train it once. Enforce everywhere. The mental model: LLMs are over-confident junior devs. You're the senior engineer. They're the force multiplier. 💾 Save this before your next AI coding session.