Innovations in AI Inference Methods

In the last three months alone, over ten papers outlining novel prompting techniques were published, boosting LLMs’ performance by a substantial margin. Two weeks ago, a groundbreaking paper from Microsoft demonstrated how a well-prompted GPT-4 outperforms Google’s Med-PaLM 2, a specialized medical model, solely through sophisticated prompting techniques. Yet, while our X and LinkedIn feeds buzz with ‘secret prompting tips’, a definitive, research-backed guide aggregating these advanced prompting strategies is hard to come by. This gap prevents LLM developers and everyday users from harnessing these novel frameworks to enhance performance and achieve more accurate results. https://lnkd.in/g7_6eP6y In this AI Tidbits Deep Dive, I outline six of the best and recent prompting methods: (1) EmotionPrompt - inspired by human psychology, this method utilizes emotional stimuli in prompts to gain performance enhancements (2) Optimization by PROmpting (OPRO) - a DeepMind innovation that refines prompts automatically, surpassing human-crafted ones. This paper discovered the “Take a deep breath” instruction that improved LLMs’ performance by 9%. (3) Chain-of-Verification (CoVe) - Meta's novel four-step prompting process that drastically reduces hallucinations and improves factual accuracy (4) System 2 Attention (S2A) - also from Meta, a prompting method that filters out irrelevant details prior to querying the LLM (5) Step-Back Prompting - encouraging LLMs to abstract queries for enhanced reasoning (6) Rephrase and Respond (RaR) - UCLA's method that lets LLMs rephrase queries for better comprehension and response accuracy Understanding the spectrum of available prompting strategies and how to apply them in your app can mean the difference between a production-ready app and a nascent project with untapped potential. Full blog post https://lnkd.in/g7_6eP6y

Reasoning Agentic RAG: The Evolution from Static Pipelines to Intelligent Decision-Making Systems The AI research community has just released a comprehensive survey that could reshape how we think about Retrieval-Augmented Generation. Moving beyond traditional static RAG pipelines, researchers from leading institutions including Beijing University of Posts and Telecommunications, University of Georgia, and SenseTime Research have mapped out the emerging landscape of Reasoning Agentic RAG. The Core Innovation: System 1 vs System 2 Thinking Drawing from cognitive science, the survey categorizes reasoning workflows into two distinct paradigms: Predefined Reasoning (System 1): Fast, structured, and efficient approaches that follow fixed modular pipelines. These include route-based methods like RAGate that selectively trigger retrieval based on model confidence scores, loop-based systems like Self-RAG that enable iterative refinement through retrieval-feedback cycles, and tree-based architectures like RAPTOR that organize information hierarchically using recursive structures. Agentic Reasoning (System 2): Slow, deliberative, and adaptive systems where the LLM autonomously orchestrates tool interaction during inference. The model actively monitors its reasoning process, identifies knowledge gaps, and determines when and how to retrieve external information. Under the Hood: Technical Mechanisms The most fascinating aspect is how these systems work internally. In prompt-based agentic approaches, frameworks like ReAct interleave reasoning steps with tool use through Thought-Action-Observation sequences, while function calling mechanisms provide structured interfaces for LLMs to invoke search APIs based on natural language instructions. Training-based methods push even further. Systems like Search-R1 use reinforcement learning where the search engine becomes part of the RL environment, with the LLM learning policies to generate sequences including both internal reasoning steps and explicit search triggers. DeepResearcher takes this to the extreme by training agents directly in real-world web environments, fostering emergent behaviors like cross-validation of information sources and strategic plan adjustment. The Technical Architecture What sets these systems apart is their dynamic control logic. Unlike traditional RAG's static retrieve-then-generate pattern, agentic systems can rewrite failed queries, choose different retrieval methods, and integrate multiple tools-vector databases, SQL systems, and custom APIs-before finalizing responses. The distinguishing quality is the system's ability to own its reasoning process rather than executing predetermined scripts. The research indicates we're moving toward truly autonomous information-seeking systems that can adapt their strategies based on the quality of retrieved information, marking a significant step toward human-like research and problem-solving capabilities.

Google's recent Gemini 2.5 report mentioned an fascinating advancement called "Deep Think" - a novel reasoning approach that enables AI models to generate multiple hypotheses in parallel and critically evaluate them before arriving at final answers. The results speak for themselves: state-of-the-art performance on challenging benchmarks including Olympiad mathematics, competitive coding, and multimodal reasoning tasks. What caught my attention was how this structured Chain-of-Thought approach could democratize advanced reasoning capabilities beyond proprietary models. So we built something similar. We developed an open-source DeepThink plugin for OptiLLM that brings these same parallel thinking techniques to open models like DeepSeek R1 and Qwen3. The plugin enables models to explore multiple solution paths simultaneously, evaluate different approaches, and converge on better answers through deeper reasoning processes. The technical implementation focuses on enhancing the reasoning pipeline during response generation, giving models the ability to internally debate and refine their approaches before presenting solutions. This is particularly valuable for complex problem-solving tasks that benefit from multi-step reasoning. We recently had the opportunity to present this work at the Cerebras Systems & OpenRouter Qwen 3 Hackathon, where it was selected as the 3rd winning project. More importantly, the plugin is now available as open source, enabling anyone to enhance their AI workflows with advanced reasoning capabilities. For those interested in the technical details, the implementation is available on GitHub at https://lnkd.in/g7nKqFt6, and I've created a demo video showing the plugin in action: https://lnkd.in/g2RwfqmC Excited to see how the community builds upon this work to advance reasoning capabilities in open AI systems. #ArtificialIntelligence #OpenSource #MachineLearning #AI #Innovation #TechLeadership

A lot has changed since my #LLM inference article last January—it’s hard to believe a year has passed! The AI industry has pivoted from focusing solely on scaling model sizes to enhancing reasoning abilities during inference. This shift is driven by the recognition that simply increasing model parameters yields diminishing returns and that improving inference capabilities can lead to more efficient and intelligent AI systems. OpenAI's o1 and Google's Gemini 2.0 are examples of models that employ #InferenceTimeCompute. Some techniques include best-of-N sampling, which generates multiple outputs and selects the best one; iterative refinement, which allows the model to improve its initial answers; and speculative decoding. Self-verification lets the model check its own output, while adaptive inference-time computation dynamically allocates extra #GPU resources for challenging prompts. These methods represent a significant step toward more reasoning-driven inference. Another exciting trend is #AgenticWorkflows, where an AI agent, a SW program running on an inference server, breaks the queried task into multiple small tasks without requiring complex user prompts (prompt engineering may see end of life this year!). It then autonomously plans, executes, and monitors these tasks. In this process, it may run inference multiple times on the model while maintaining context across the runs. #TestTimeTraining takes things further by adapting models on the fly. This technique fine-tunes the model for new inputs, enhancing its performance. These advancements can complement each other. For example, an AI system may use agentic workflow to break down a task, apply inference-time computing to generate high-quality outputs at each step and employ test-time training to learn unexpected challenges. The result? Systems that are faster, smarter, and more adaptable. What does this mean for inference hardware and networking gear? Previously, most open-source models barely needed one GPU server, and inference was often done in front-end networks or by reusing the training networks. However, as the computational complexity of inference increases, more focus will be on building scale-up systems with hundreds of tightly interconnected GPUs or accelerators for inference flows. While Nvidia GPUs continue to dominate, other accelerators, especially from hyperscalers, would likely gain traction. Networking remains a critical piece of the puzzle. Can #Ethernet, with enhancements like compressed headers, link retries, and reduced latencies, rise to meet the demands of these scale-up systems? Or will we see a fragmented ecosystem of switches for non-Nvdia scale-up systems? My bet is on Ethernet. Its ubiquity makes it a strong contender for the job... Reflecting on the past year, it’s clear that AI progress isn’t just about making things bigger but smarter. The future looks more exciting as we rethink models, hardware, and networking. Here’s to what the 2025 will bring!

1/ SakanaAI just dropped their latest research: Transformer2 It's a self-adaptive architecture that allows AI to evolve at inference time. Model weights are no longer "static" Let’s break it down: 🧵 2/ Traditional Transformers are static post-training. Once trained, they can’t learn or adapt without expensive fine-tuning or additional methods like retrieval-augmented generation (RAG). Transformer2 changes this entirely. 3/ The core innovation? A two-pass system. 🌀 • Pass 1: Analyze the task (e.g., math, coding, or reasoning) to understand the query. • Pass 2: Dynamically update specific model weights based on the task. This makes the model far more adaptable. 4/ Transformer2 uses Selective Weight Updating Only adjusting task-relevant weights during inference. This is super efficient and avoids the costs of traditional fine-tuning while enabling real-time learning. 5/ The key method behind this is Singular Value Fine-Tuning (SVF): • It adjusts specific components of the model’s weight matrices. • Think of it as a "surgical" approach to fine-tuning – precise, efficient, and effective. 6/ Why does this matter? 🤔 • Models can continuously improve at inference time without retraining. • They handle diverse tasks dynamically, adapting in real time. • Open-source accessibility makes it easier for the community to experiment and innovate. (link down below!) 7/ SakanaAI also highlights how this mimics human cognition. 🧠 Just like our brain activates specific regions for different tasks (e.g., math vs. writing), Transformer2 uses modular "expert vectors" for task-specific adjustments. 8/ Results? 🚀 Transformer2 outperforms traditional methods like LoRA in efficiency and accuracy. It achieves better results with fewer parameters and less resource usage – an exciting step forward for scalable AI. 9/ More results! And it’s not limited to language tasks. Transformer2 also works well for vision models, demonstrating its versatility across different domains. 10/ What’s next? SakanaAI’s open-sourced code lets anyone explore this technology today. 🌐 This could be a major leap for AI, bridging the gap between static models and dynamic, ever-evolving systems. Is this a new scaling law? 11/ Links! Check out the paper here: https://lnkd.in/dU2KKJTi The open-source code here: https://lnkd.in/dnN82KVp And my full video breakdown here: https://lnkd.in/d5dcCjA7

Why Compound AI Systems Are Taking Over ⭐ We’re moving beyond single-model AI into an era where Compound AI Systems—modular, flexible, and powerful—are setting a new standard. But what does this mean? And why should AI leaders pay attention? 🔍 𝗪𝗵𝗮𝘁 𝗔𝗿𝗲 𝗖𝗼𝗺𝗽𝗼𝘂𝗻𝗱 𝗔𝗜 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 Unlike traditional AI models that work in isolation, Compound AI Systems integrate multiple components—LLMs, retrieval mechanisms, external tools, and reasoning engines—to solve complex problems more effectively. Instead of relying on one massive model, these systems: ✔️ Combine multiple AI models for specialized tasks ✔️ Use retrieval mechanisms to fetch real-time, relevant data ✔️ Leverage external tools (APIs, databases, or symbolic solvers) to enhance reasoning ✔️ Improve adaptability by dynamically selecting the best approach for a given problem This modular approach enhances accuracy, efficiency, and scalability—giving AI systems the ability to think beyond their training data and operate more intelligently in real-world environments. 🏆 𝗪𝗵𝗲𝗿𝗲 𝗖𝗼𝗺𝗽𝗼𝘂𝗻𝗱 𝗔𝗜 𝗜𝘀 𝗪𝗶𝗻𝗻𝗶𝗻𝗴 ↳ Google’s AlphaCode 2 Generates millions of programming solutions, then intelligently filters out the best ones—resulting in dramatic improvements in AI-driven code generation. ↳ AlphaGeometry Combines a large language model (LLM) with a symbolic solver, enabling AI to solve complex geometry problems at an expert level. ↳ Retrieval-Augmented Generation (RAG) Now a standard in enterprise AI, RAG models retrieve relevant data in real-time before generating responses, significantly boosting accuracy and contextual relevance. ↳ Multi-Agent Systems Startups and research labs are developing AI "teams"—where multiple models communicate and collaborate to solve problems faster and more efficiently than a single model could. 💡 𝗪𝗵𝘆 𝗜𝗻𝗱𝘂𝘀𝘁𝗿𝘆 𝗟𝗲𝗮𝗱𝗲𝗿𝘀 𝗔𝗿𝗲 𝗕𝗲𝘁𝘁𝗶𝗻𝗴 𝗕𝗶𝗴 𝗼𝗻 𝗖𝗼𝗺𝗽𝗼𝘂𝗻𝗱 𝗔𝗜 This isn’t just a research trend. It’s an industry-wide shift. ↳ Microsoft, IBM, and Databricks are already pivoting their AI strategies toward modular, system-based AI architectures. ↳ Fireworks AI is leading the GenAI inference platform with Compound AI Systems ↳ Even OpenAI’s CEO, Sam Altman, emphasized the transition: "We’re going to move from talking about models to talking about systems." 𝗧𝗵𝗲 𝗕𝗶𝗴 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆 𝗳𝗼𝗿 𝗔𝗜 𝗟𝗲𝗮𝗱𝗲𝗿𝘀 The implications are massive: ✔️ AI performance will increasingly depend on system design—not just model size ✔️ Custom AI solutions will become the norm, allowing businesses to tailor AI systems for specific needs ✔️ Efficiency will skyrocket, as compound systems reduce computational waste by dynamically choosing the best approach for a given task ----------------------- Share this with your network ♻️ Follow me (Aishwarya Srinivasan) for more AI insights, news, and educational resources to keep you up-to-date about the AI space!

Deepseek has disrupted AI across many dimensions. One of them is doing more work during inference at the expense of training. Increasing inference time compute seems like an intuitive “Pay per View” tradeoff and other models are rapidly adopting it. Inference workloads already use twice as much of computing as training. The balance is likely to tilt more that way. Would this make a measurable impact on data center hardware mix? Inference workloads are distributed across a greater diversity of hardware platforms such as CPU, GPU, FPGA, ASIC, each of which exhibit distinct characteristics which can help improve LLM inference performance. CPUs excel in programmability, GPUs have massive parallel capabilities and memory bandwidth, FPGAs and ASICs are often designed for specific applications with the customized architecture offering higher computational efficiency and better energy efficiency. Different hardware platforms may also be combined to generate optimum tradeoffs between performance, accuracy, power, and cost. Optimizations for inference can also offer tradeoffs. These include quantization of weights and values (integer vs. float, bits of precision), selection of different evaluation operators (linear vs. non-linear), short-cuts like skipping layers in the model etc. There are four times as many CPUs shipped annually into data centers than GPUs. The installed base of CPUs is also much larger. Undoubtedly efficient use of capital favors using CPUs as much as prudent for inference. AMD, the current darling of server CPUs says this: “A powerful foundation for AI workflows, 5th Gen AMD EPYC processors are the ideal CPU-based AI platform to run inference across a variety of models and use cases. (They) deliver the flexibility to support requirements ranging from real time inference to batch or offline inference.” Investors seem bullish on CPUs even when (unlike x86 server chips) current shipments and installed base are miniscule. Softbank recently announced the acquisition of Ampere - one of the only merchant vendors of ARM based CPU server chips. AheadComputing a startup by Intel veterans secured healthy seed funding to rapidly develop and commercialize breakthrough RISC-V microprocessor architecture for computing demands across AI, cloud, and edge devices. Industry veterans like Jim Keller and David Ditzel are heading companies working on chips for AI that are substantially based on CPU and CPU performance. The Deepseek disruption is remarkable for arriving so early in the AI cycle. It has focused minds on reducing the end user’s cost for profitably using and deploying AI widely. This mission will have major implications for AI hardware as well. The CPU emerged as the Swiss army knife of computing after the first microprocessor on a chip was designed instead of an ASIC for a business calculator 50 years ago. Its Swiss army knife value might continue to help it thrive through the AI revolution. 

NVIDIA did it again - 5-6x faster generation than the current standard. TiDAR from NVIDIA is a new hybrid architecture that achieves what seemed impossible: 5-6× faster generation than standard autoregressive models while maintaining the same output quality. The fundamental challenge with current LLMs is computational efficiency. Autoregressive models like GPT generate one token at a time, leaving GPU compute largely underutilized in a memory-bound regime. Diffusion language models can generate multiple tokens in parallel, but this comes at a steep cost - quality degrades significantly when attempting to decode more than one token per step. TiDAR resolves this tension through a dual-mode architecture. The model "thinks" in diffusion by drafting multiple tokens in parallel, then "talks" in autoregression by sampling final outputs through rejection sampling - all within a single forward pass. The key innovation is a carefully designed attention mask that enables both causal and bidirectional attention patterns simultaneously. This allows the model to leverage what the researchers call "free token slots" - additional computation that incurs minimal latency cost in the memory-bound regime. TiDAR generates 7-8 tokens per forward pass, translating to 4.71x to 5.91x throughput improvements measured in tokens per second. Most importantly, it's the first architecture to close the quality gap with pure autoregressive models while delivering these speedups. The method even outperforms EAGLE-3, currently the leading speculative decoding approach, in both throughput and quality metrics across coding, math, and reasoning tasks. This could lead to a fundamental shift in LLM inference optimization. Rather than choosing between quality and speed, or relying on separate draft models with limited capacity, TiDAR shows we can achieve both by intelligently combining complementary sampling strategies within a unified architecture. ↓ 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

🔍 ϕ-Decoding: New Sampler for Reasoning This paper introduces a novel inference-time optimization algorithm that improves LLM reasoning without additional training by balancing exploration and exploitation during the decoding process. → The authors frame decoding as "foresight sampling" - using simulated future steps to estimate global optimal reasoning paths while avoiding the computational expense of extensive tree search methods. → ϕ-Decoding combines two distributions for step value estimation: one based on step advantage values (uncertainty differences between steps) and another from clustering foresight paths to assess their alignment. → Dynamic pruning strategies (in-width and in-depth) intelligently allocate computational resources, dedicating more to early critical reasoning steps while reducing "overthinking" in later steps. → Experimental results show significant performance improvements (14.62% on LLaMA3.1-8B) across diverse reasoning benchmarks while maintaining better efficiency than baseline methods (6x faster for comparable performance). It's nice to see more work around sampling+reasoning. This was a popular topic last year around September and October, but turned out to be another overhyped thing in the AI community. I wonder about diminishing returns at larger compute budgets and whether the approach would benefit from incorporating external knowledge sources for complex problems. The most promising aspect is how it reduces "overthinking" (a common inefficiency in LLM reasoning processes), which could significantly impact real-world inference costs for reasoning-heavy applications.

Chain of Thought might not be the future after all... This super interesting paper introduces a new prompting technique—Chain of Draft (CoD)—that improves efficiency by making AI models think faster while writing less. We know Chain of Thought (CoT) has been a major breakthrough, guiding models to reason step by step. But it comes with trade-offs: CoT generates a lot of extra tokens, increasing cost and slowing inference. CoD takes a different approach. Instead of making the model write out full explanations, it encourages concise, structured reasoning—more like shorthand notes than long essays. And the results look amazing: 🔹Tested on GPT-4o and Claude 3.5 → Similar or better accuracy than CoT 🔹 80–92% fewer tokens depending on the task → Lower cost, faster inference 🔹 No fine-tuning required → Just a better prompt We’ve seen other techniques aimed at optimizing AI efficiency—like Skeleton of Thought, which first generates an outline before expanding. CoD is even simpler: It doesn’t change how AI reasons, just how much it writes down.💡 With inference cost and latency becoming major bottlenecks—especially for real-time AI like chatbots, on-device models, and large-scale deployments—efficient prompting strategies like CoD are worth paying attention to. What do you think? Could this become standard practice in AI inference? __________ For more on AI, please check my previous posts. I share my journey here. Join me and let's grow together. Alex Wang #ai #llms #machinelearning #generativeai