Goku Mohandas

Google released a new 270M Gemma 3 variant last week, a lightweight, open-weight LLM that's perfect for local fine-tuning experiments and general tinkering. I couldn't resist and re-implemented it from scratch in pure PyTorch: https://lnkd.in/gSGzZvZv A few interesting tidbits about the Gemma 3 architecture: 1) One of the few recent architectures (besides gpt-oss) that uses Sliding Window Attention with a ratio of 5 local to 1 global layers 2) Dual RMSNorm: unlike standard Pre-Norm or Post-Norm transformers, Gemma 3 uses RMSNorm both before and after its attention + MLP blocks (I added the Qwen 3 figure showing the standard placement for comparison) 3) Vocabulary size: similar to other Gemma 3 models, it uses a huge vocabulary (~256k tokens), which should help with better multilingual coverage The code in my notebook requires about 1.49 GB of RAM (for inference) and runs comfortably on a laptop / Mac Mini. (Note that while it has good instruction-following capabilities given its small size, it's of course not a go-to model for local inference; the focus is more on being fast and lightweight for research and experimentation purposes.) Happy tinkering!

4,528

132 Comments

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Today, let's learn: - Function calling by building custom tools for Agents. - How MCPs help by building a local MCP client with mcp-use and using tools from Browserbase MCP server. In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point: MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable, without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. In this setup, to build a local MCP client, I used mcp-use because it lets us connect any LLM to MCP servers & build private MCP clients, unlike Claude/Cursor. - Compatible with Ollama & LangChain - Stream Agent output async - Built-in debugging mode, etc Find the mcp-use GitHub repo in the comments! ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

269

11 Comments

🚀 Fast-dLLM v2: Parallel Block-Diffusion Decoding for LLMs ⚡️ Highlights 🌟   - Blockwise bidirectional context via complementary masks   - Hierarchical caches (block + sub-block)   - Parallel sub-block decoding + token-shift training Results 📊 - ~2.5× faster vs. standard AR decoding on A100   - 102.5 tok/s (bs=1), 201.0 tok/s (bs=4) — >2× Qwen2.5   - SOTA efficiency–quality trade-off among diffusion LLMs Paper/Code 🌐: https://lnkd.in/g3AhGp96

264

You're in a ML Inference Engineer interview at NVIDIA, and the interviewer asks: "Our team wants to switch from API to a self-hosted. Which serving framework and why?" Here's how you answer: Don't say: "vLLM is fastest" or "Ollama is easiest." Wrong framing. The real answer isn't about features - it's about matching serving philosophy to your constraints. Local prototype vs. production scale vs. complex workflows = completely different frameworks. NOTE - if you want to read this kinda content daily, consider subscribing my free newsletter - https://lnkd.in/g8ZJGsWj Most teams pick based on benchmarks, not bottlenecks Your serving choice matters less than your use case. PagedAttention isn't magic - it's just better memory management "Fastest" framework can be slowest for YOUR workload The infrastructure reality everyone misses: Single user prototype = Ollama/LLaMA.cpp wins Multi-user production = vLLM dominates AI agent workflows = SGLang is your only option Edge deployment = LLaMA.cpp or bust Architecture drives the decision, not features. "But what about performance?" Interviewer: "How do you handle 1000 concurrent users?" Performance without context is meaningless. > vLLM's PagedAttention gives you 24x throughput, but only if memory fragmentation was your bottleneck. > SGLang's RadixAttention only helps with cache reuse scenarios. By the way, just reading about inference frameworks won't help you land a job. I personally learnt a lot by doing stuff. If I were to do all over, I would rent a gpu from Hyperbolic for dirt cheap, and play with some open source model with different frameworks. The business framework that matters: > Rapid prototyping + Local dev = Ollama > Known high load + GPU budget = vLLM > Complex multi-step LLM workflows = SGLang > Constrained environments + Portability = LLaMA.cpp Match the tool to the business constraint. The evolution path most teams miss: > Start: Ollama (prototype fast) > Scale: vLLM (handle users) > Optimize: SGLang (complex workflows) > Edge: LLaMA.cpp (deploy anywhere) It's not replacement - it's graduation based on needs. The answer that gets you hired: "Framework choice isn't about better vs worse. It's about constraints vs capabilities. Ollama gives you speed-to-market. vLLM gives you user scale. SGLang gives you workflow complexity. Pick based on your biggest constraint, not your biggest benchmark." That's it for today. See ya tomorrow! And if you want to read more about these frameworks, checkout this blog: https://lnkd.in/eiH2sphA #llm #machinelearning #gpu #chatgpt #ai #agents

398

17 Comments

𝗧𝗵𝗲 𝗱𝗮𝘆 Yann LeCun 𝗿𝘂𝗶𝗻𝗲𝗱 𝘀𝗰𝗮𝗹𝗮𝗿 𝗯𝗮𝗰𝗸𝗽𝗿𝗼𝗽 𝗳𝗼𝗿 𝗺𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿 2013. NYU. I'm sitting in Yann's class during my post-doc, struggling with backprop implementation for the nth time. I'd been trying to code it up the "traditional" way - tracking individual parameters, computing gradients one by one. My notebooks looked like a crime scene of subscripts and indices. Then Yann draws this on the board: 𝗧𝗵𝗶𝗻𝗸 𝗶𝗻 𝘃𝗲𝗰𝘁𝗼𝗿𝘀. 𝗡𝗼𝘁 𝘀𝗰𝗮𝗹𝗮𝗿𝘀. Wait, what? "Your activations are vectors. Your weights are vectors. Your gradients? Also vectors. Why are you torturing yourself with individual elements?" Here's what blew my mind: - When your output is a column vector, the gradient is a 𝗿𝗼𝘄 𝘃𝗲𝗰𝘁𝗼𝗿 of the 𝗲𝘅𝗮𝗰𝘁 𝘀𝗮𝗺𝗲 𝘀𝗶𝘇𝗲 - The scary "Jacobian"? Just a vector differentiated by another vector - literally just all the partial derivatives arranged in a matrix - No magic. Just organization. In the 59-second video above, I'm distilling what Yann taught me that day. It changed everything. Instead of: - ∂L/∂w_ij for every single weight - Tracking which activation connects to which - Getting lost in notation hell You get: - Clean matrix vector operations - Each module handles its own gradient computation - Code that looks like the math The real kicker? Once you see it this way, you can build your own PyTorch. No joke. Each module just needs two things: 1. 𝗙𝗼𝗿𝘄𝗮𝗿𝗱: input → output 2. 𝗕𝗮𝗰𝗸𝘄𝗮𝗿𝗱: gradient_out → gradient_in, gradient_weights Chain them together. Boom. Deep learning library. This insight didn't just help me implement backprop. It got me onto the core #Theano development team (yes, the paper with 1000s of citations). Because when you truly understand the fundamentals, you can build the tools everyone else uses. 𝗛𝗲𝗿𝗲'𝘀 𝘄𝗵𝗮𝘁 𝗻𝗼𝗯𝗼𝗱𝘆 𝘁𝗲𝗹𝗹𝘀 𝘆𝗼𝘂: The "harder" mathematical view is actually easier. Vector calculus isn't showing off - it's simplifying. To everyone still wrestling with scalar backprop: Stop. Watch the video. Think in vectors. Stop fighting the math. Let it work for you. Your future self will thank you. Drop a 🔥 if you're ready to finally understand backprop the right way. #DeepLearning #MachineLearning #Backpropagation #AI #Engineering

1,762

101 Comments

🔥 Cool paper dropped today: “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning” by Lakshya A Agrawal, Omar K. and others. Instead of using RL’s scalar rewards, GEPA leverages natural language feedback from system traces to reflectively evolve prompts. They got ~10% better performance than GRPO with 35x fewer rollouts. This supports a broader point that we’ve been focused on at Veris AI: when you have an environment that provides rich feedback, RL is just one optimization tool in your toolbox. The magic happens when your environment generates high quality structured, interpretable feedback that can be consumed in multiple ways. You don’t have to just collapse them into scalar rewards. Smart system design = flexible feedback mechanisms that unlock diverse optimization strategies. Side note: Berkeley folks have been killing it recently with everything going on at Skylab and BAIR. Go bears.

68

4 Comments

Investment is moving from pre-training to post-training. Early LLM budgets focused on training ever-larger base models. Post-deployment, models were mostly fixed... improvements came via scaling next-gen models. Now labs split resources between pre-training and RL fine-tuning with careful inference optimization. In post-training, models learn reasoning, follow prefs and refine outputs for specific tasks. Models update more frequently, weekly or monthly vs yearly. Each iteration collects new data, extends RL training and adjusts rewards to improve reasoning. New economic models charge for updated reasoning engines vs static models. Serving costs and latency are key so providers favor efficient pro models with most accuracy at lower compute. 3 more takeaways from ICML: 1️⃣ RL is only beginning to cross into “unverifiable” domains. Traditional RL used tasks with clear auto checks (like code compilers or math calcs) for rewards. Domains like Math Olympiad or legal arguments have complex solutions that can't be auto-verified. These need more complex reward models scoring reasoning steps, argument clarity or persuasiveness vs just correctness. With solid reward models, RL could teach coherent proofs, experiment design or legal strategies. This is still early work - current systems often break tasks into verifiable parts or use human prefs, but broader paths are emerging. 2️⃣ Personalization is a safer near-term goal than true continual learning. Interest in adapting models to users is growing with 2 key approaches. Personalization tailors outputs by learning user-specific rewards from few feedbacks or adding curiosity rewards that prompt questions about user tastes. These adjust behavior without changing model weights, improving perceived helpfulness and empathy. Continual learning updates parameters in real time from all user input. But continual learning poses safety and privacy risks like overfitting, bias amp & data leakage. Personalization and context windows (remembering interactions without weight changes) are more practical and responsible. 3️⃣ RL scaling will follow multiple paths. → One path applies current RL methods (PPO, guided reward models, diffusion reg) to more domains. This incremental path needs no new algos, just better rewards, bigger varied datasets & optimization tweaks. → The second tackles sparse-reward probs with long feedback delays. Success depends on credit assignment in long seqs and off-policy learning using varied experiences. Off-policy RL supports multi-datacenter training with clusters handling acting, collection and learning. → The third involves continual learning with frequent updates using user feedback and new data. Each has trade-offs: incremental and safe, risky but domain-expanding, adaptive but complex w/ safety issues. What'd I miss?

50

5 Comments

Stop chasing FLOPs. The real gains in AI inference are elsewhere. SemiAnalysis team and their InferenceMax report : The next leap in inference efficiency = hardware × software × orchestration Running Llama 3.3 70B FP8 across (AMD and NVIDIA) GPUs 1. It’s memory-bound, not compute-bound. H200 and B200 outperform older chips because bandwidth and interconnects matter more than peak FLOPs. 2. Software doubles the value of hardware. TensorRT + FP8 nearly doubles throughput—stack optimization now equals silicon innovation. 3. Serving is the new bottleneck. Multi-user loads flatten curves; smarter batching and schedulers are the next frontier..

271

8 Comments

Introducing SemTools - add blazing-fast semantic search to your entire filesystem without a vector database ⚡️ Coding agents like Claude Code/Cursor have full access to the CLI like grep, cat, and pipe operations for search. But they lack ‘proper` semantic search that’s actually fast. SemTools adds in two additional CLI commands: 1️⃣ `parse` for document parsing 2️⃣ `search` to dynamically chunk/embed/search in-mem based on any set of files using static embeddings (400x faster than dense). You can chain these commands with existing operations like grep + pipes. Importantly, you also don’t need to maintain a vector db or dense heavyweight embedding models - these are overkill, slower, and probably lower accuracy vs. just a CLI agent which has access to other CLI operations too. (See Claude code screenshot below with semtools in action - see `parse` and `search` commands used) This means you can easily give 1k+ PDFs/other docs within your filesystem to your favorite coding agent/general agent, and have it efficiently crunch through any context to pull in just what the agent needs. Huge shoutout to Logan Markewich. Check it out: https://lnkd.in/gicRBe9D

916

46 Comments

My colleague and friend Hartmut Neven once told me the best way to think about quantum computing was to imagine chopping an onion. You could do it the ‘traditional’ way: peeling the onion, later by layer, and chopping each layer in turn. That’s a classical computer.  But what if there was a different way? What if you could hold the whole onion and run your knife through it at once, getting the result but so much faster. That’s a quantum computer. Right now, ‘doing the whole onion' means quantum computer chips that can complete computations in under 5 minutes that would take today’s fastest supercomputers 10 septillion years. (To save you a Google: a septillion has 24 zeros ) When you combine computers of that speed and scale with AI, the possibilities are endless - from discovering new medicines quicker, to accelerating new types of energy like fusion. That’s the potential the Nobel judges recognised yesterday, when they awarded Google Quantum AI's Michel Devoret, his former Google colleague John Martinis, and Berkeley’s John Clarke the 2025 Nobel Prize for Physics for their pioneering discoveries in quantum. We already have their breakthroughs to thank for technologies we use every day, from mobile phones to cameras. Now they’re steering us towards the future with quantum - and it’s incredibly exciting to think that these are still just the early days. Huge congratulations to Michel, John and John! More in Harmut's blog: https://lnkd.in/dx5havXE (And glad to have the chance to revisit this photo with Hartmut, Demis and Eileen 🙂)

170

9 Comments

RAG systems live or die by the quality of their retrieval pipeline. If the right context doesn’t surface, even the most advanced LLM will stumble. As Google DeepMind’s recent paper highlights, embedding-based retrieval runs into surprising limitations. Even with a modest corpus of ~10,000 chunks, vanilla embedding search can fail to fetch the most relevant passages. So, what’s the way forward? Retrieval needs to get smarter. In this post, Debjit breaks down three powerful techniques that address these shortcomings and significantly boost retrieval quality: hybrid search, reranking, and multi-vector retrieval. Let’s explore how these optimizations transform RAG from “hit-or-miss” to consistently reliable. Full read here - https://lnkd.in/dVfEyY4S

35

1 Comment

An interesting new paper on LLM-JEPA from Hai Huang, Yann LeCun, and Randall Balestriero. 💡 Previously I wrote about JEPA approach applied to videos (V-JEPA and V-JEPA 2) and time series (CHARM). Now the JEPA approach is finally applied to LLMs! This work bridges a major gap between AI for vision and language, offering a potential leap forward in how we train language models. Instead of just predicting the next word, LLM-JEPA teaches models to understand the underlying meaning by predicting abstract concepts (as JEPA approach does)—for instance, grasping the essence of a code snippet from its natural language description. The paper introduces a hybrid objective combining standard next-token prediction with a Joint Embedding Predictive Architecture (JEPA) loss, a technique highly successful in computer vision. The empirical results are compelling: LLM-JEPA consistently boosts performance, accelerates parameter-efficient fine-tuning (PEFT), and shows remarkable resistance to overfitting. This method doesn't just improve scores; it fundamentally creates more structured and transferable representations. While the current computational overhead is a challenge to address, this paper opens a promising new direction beyond traditional LLM training. 🚀 Review: https://lnkd.in/eC4Jte_r Paper: https://lnkd.in/erZJadb3 Code: https://lnkd.in/ethXT7sX

73

3 Comments