Harshit Sharma

Google released a new 270M Gemma 3 variant last week, a lightweight, open-weight LLM that's perfect for local fine-tuning experiments and general tinkering. I couldn't resist and re-implemented it from scratch in pure PyTorch: https://lnkd.in/gSGzZvZv A few interesting tidbits about the Gemma 3 architecture: 1) One of the few recent architectures (besides gpt-oss) that uses Sliding Window Attention with a ratio of 5 local to 1 global layers 2) Dual RMSNorm: unlike standard Pre-Norm or Post-Norm transformers, Gemma 3 uses RMSNorm both before and after its attention + MLP blocks (I added the Qwen 3 figure showing the standard placement for comparison) 3) Vocabulary size: similar to other Gemma 3 models, it uses a huge vocabulary (~256k tokens), which should help with better multilingual coverage The code in my notebook requires about 1.49 GB of RAM (for inference) and runs comfortably on a laptop / Mac Mini. (Note that while it has good instruction-following capabilities given its small size, it's of course not a go-to model for local inference; the focus is more on being fast and lightweight for research and experimentation purposes.) Happy tinkering!

4,528

132 Comments

Just getting started with LLMs? Don’t let the jargon slow you down. This 5-part LLM Glossary is your shortcut to mastering the fundamentals👇 1️⃣ Model Types • Foundation Model: Pretrained on large-scale, diverse datasets to learn general capabilities. • Instruction-Tuned Model: Further trained to follow user instructions precisely. • Multi-modal Model: Handles multiple input/output formats like text, images, and audio. • Reasoning Model: Optimized for logic, problem-solving, and step-by-step thinking. • Small Language Model (SLM): Lightweight models for fast, efficient, on-device tasks. 2️⃣ Training & Fine-Tuning • Pretraining: Initial learning phase using massive, diverse data sources. • RLHF (Reinforcement Learning with Human Feedback): Aligns model behavior with human preferences. • DPO (Direct Preference Optimization): Uses ranked preferences instead of rewards for training. • Synthetic Data: AI-generated data used to supplement real datasets. • Fine-Tuning: Targeted retraining on specific domains or tasks. • LoRA / QLoRA: Techniques for efficient low-resource fine-tuning. • Guardrails: Rules or filters applied to enforce safety, ethics, and compliance. 3️⃣ Prompt Engineering • System/User Prompts: Define model behavior and user context in a session. • Chain-of-Thought (CoT): Prompts that guide the model to reason step-by-step. • Few-Shot / Zero-Shot Learning: Demonstrating tasks with few or no examples. • Prompt Tuning: Optimizing prompts via training for specific outcomes. • Context Window: The total number of tokens the model can "remember" per interaction. 4️⃣ Inference & Generation • Temperature: Adjusts output randomness; lower values are more deterministic. • Max Tokens: Limits the number of tokens in a generated response. • Seed: Controls reproducibility of outputs. • Latency: Time taken by the model to respond. • Hallucination: When the model generates factually incorrect but plausible-sounding information. 5️⃣ Retrieval-Augmented Generation (RAG) • Retrieval: Fetching relevant data from external sources before generation. • Semantic Search: Search based on meaning, not just keywords. • Embeddings: Vector representations of text used for similarity matching. • Chunks: Dividing documents into smaller segments for better retrieval. • Vector Databases (VectorDBs): Stores embeddings for fast and accurate search. • Reranking: Rescoring retrieved documents to prioritize relevance. • Indexing: Structuring data for efficient retrieval during generation. 🌟 Follow the AIKaDoctor (Free AI & Data Science Resources) channel on WhatsApp: https://lnkd.in/dCTCEKKc 📌Follow Habib Shaikh For more such content.

232

36 Comments

🚀 Context Engineering: The Science Behind Smarter LLMs 📄 Just finished reading “A Survey of Context Engineering for Large Language Models” — and it’s a game-changer for anyone working with or building LLM systems. 💡 Key Takeaways: 🔹 Context Engineering is more than prompt design. It’s the formal optimization of the information payload that LLMs consume — including retrieval, formatting, memory, and orchestration. 🔹 A Unified Framework: The paper breaks down Context Engineering into 3 foundational components: 1. Context Retrieval & Generation – e.g., Prompting, RAG, Dynamic Assembly 2. Context Processing – e.g., Long-sequence modeling (Mamba, FlashAttention), Self-Refinement (Reflexion, Self-RAG) 3. Context Management – e.g., Compression (H2O, QwenLong), Memory Architectures (MemGPT, MemoryBank) These are then integrated into powerful systems: • RAG 2.0+: Agentic RAG, GraphRAG, Modular RAG • Memory-Augmented Agents: Like MemOS and RecMind • Tool-Integrated Reasoning: Think Toolformer, AutoGen • Multi-Agent Systems: CAMEL, SagaLLM, CrewAI 🔍 Why it matters: • 18x gains in information navigation • Up to 94% task success rates using context optimization • Models learn, reflect, and self-improve without retraining • Enables real-time tool use and long-term memory across tasks ⚠️ Biggest Challenge Identified: Despite advances in understanding complex context, LLMs still struggle to generate equally complex, structured long-form outputs. Bridging this gap is a priority for future research. 🌐 What’s Next? • Scalable architectures with context compression • Unified benchmarks for evaluating tool use and memory • Human-AI collaboration powered by multimodal, structured, and self-optimizing systems If you’re building context-aware LLMs, designing retrieval strategies, or exploring multi-agent orchestration — this paper is a must-read. 🔗 arXiv Link: https://lnkd.in/efyyXGQz 📚 GitHub Resource: https://lnkd.in/eiqdA4Dw #LLM #AI #ContextEngineering #RAG #Agents #Toolformer #AIResearch #NLP #MemorySystems #MultiAgentSystems #PromptEngineering #SelfRefinement #GenerativeAI

166

5 Comments

Stop chasing FLOPs. The real gains in AI inference are elsewhere. SemiAnalysis team and their InferenceMax report : The next leap in inference efficiency = hardware × software × orchestration Running Llama 3.3 70B FP8 across (AMD and NVIDIA) GPUs 1. It’s memory-bound, not compute-bound. H200 and B200 outperform older chips because bandwidth and interconnects matter more than peak FLOPs. 2. Software doubles the value of hardware. TensorRT + FP8 nearly doubles throughput—stack optimization now equals silicon innovation. 3. Serving is the new bottleneck. Multi-user loads flatten curves; smarter batching and schedulers are the next frontier..

271

8 Comments

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Today, let's learn: - Function calling by building custom tools for Agents. - How MCPs help by building a local MCP client with mcp-use and using tools from Browserbase MCP server. In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point: MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable, without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. In this setup, to build a local MCP client, I used mcp-use because it lets us connect any LLM to MCP servers & build private MCP clients, unlike Claude/Cursor. - Compatible with Ollama & LangChain - Stream Agent output async - Built-in debugging mode, etc Find the mcp-use GitHub repo in the comments! ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

269

11 Comments

🚀 New blog post: How we're using multiple LLMs (Gemini, Claude, ChatGPT & more) to revolutionize software performance optimization at Fastcode ✨ Feeding academic papers + source code into AI models to uncover optimization opportunities that would take months to find manually 📈 Results: ~67% of AI recommendations lead to measurable performance gains 🔗 https://bit.ly/45WGw6u #AI #PerformanceEngineering #LLMs #SoftwareOptimization #Fastcode

65

3 Comments

Strong start yesterday — great mix of Datacolor AI engineers, UTD students, and AWS/Informatica mentors diving deep into what Agentic AI really means at an enterprise scale. We spent the day iterating on technical patterns, how to build multi-agent orchestration pipelines ,use AWS Bedrock, Claude, and LangGraph, while linking to Informatica’s metadata and data-governance APIs for trusted, context-aware execution. Students from The University of Texas at Dallas brought serious curiosity — some experimenting with the Model Context Protocol (MCP) runtime to connect agents to real business systems For us, this hackathon isn’t about demos — it’s about giving back: sharing frameworks, mentoring students, and helping build the next generation of AI engineers who understand both the model and the system.

70

5 Comments

Some signs that Waymo has jumped on the end-to-end machine learning bandwagon. Junko Yoshida has a scoop from attending The Autonomous: signs that Waymo is doing tentative deployments of end-to-end machine learning in their fleet. Complete with a quote from Waymo that seems to support this (or at least does not deny it). Such a change has deep implications for scalability as well as safety. More about this in the article. I fully expect that Waymo will be smarter than a simplistic pixels-in/driving-out approach, as indicated in the story. But what that will look like is speculation at this point. https://lnkd.in/e4vTmRTs

88

9 Comments

Recently Meta made headlines with unprecedented, massive compensation packages for AI model builders exceeding $100M (sometimes spread over multiple years). With the company planning to spend $66B-72B this year on capital expenses such as data centers, a meaningful fraction of which will be devoted to AI, from a purely financial point of view, it’s not irrational to spend a few extra billion dollars on salaries to make sure this hardware is used well. A typical software-application startup that’s not involved in training foundation models might spend 70-80% of its dollars on salaries, 5-10% on rent, and 10-25% on other operating expenses (cloud hosting, software licenses, marketing, legal/accounting, etc.). But scaling up models is so capital-intensive, salaries are a small fraction of the overall expense. This makes it feasible for businesses in this area to pay their relatively few employees exceptionally well. If you’re spending tens of billions of dollars on GPU hardware, why not spend just a tenth of that on salaries? Even before Meta’s recent offers, salaries of AI model trainers have been high, with many being paid $5-10M/year, although Meta has raised these numbers to new heights. Meta carries out many activities, including run Facebook, Instagram, WhatsApp, and Oculus. But the Llama/AI-training part of its operations is particularly capital-intensive. Many of Meta’s properties rely on user-generated content (UGC) to attract attention, which is then monetized through advertising. AI is a huge threat and opportunity: If AI-generated content (AIGC) substitutes for UGC to capture people's attention to sell ads against, this will transform the social-media landscape. This is why Meta — like TikTok, YouTube, and other social-media properties — is paying close attention to AIGC, and why making significant investments in AI is rational. Further, when Meta hires a key employee, not only does it gain the future work output of that person, but it also gets insight into a competitor’s technology, which also makes its willingness to pay high salaries a rational business move (so long as it does not adversely affect the company’s culture). The pattern of capital-intensive businesses compensating employees extraordinarily well is not new. For example, Netflix expects to spend a huge $18B this year on content. This makes the salary expense of paying its 14,000 employees a small fraction of the total expense, which allows the company to routinely pay above-market salaries. Its ability to spend this way also shapes a distinctive culture that includes elements of “we’re a sports team, not a family” (which seems to work for Netflix but isn’t right for everyone). In contrast, a labor-intensive manufacturing business like Foxconn, which employs over 1 million people, has to be much more price-sensitive in what it pays people. [Continued in comment below; truncated here due to length limit. Original: https://lnkd.in/gSGpymM5 ]

2,398

185 Comments

Memory in LLMs functions as the substrate that determines whether these systems remain parlor tricks or evolve into enduring collaborators. Too often, the conversation collapses into “context windows” or “retrieval hacks,” as if memory equates to more tokens or smarter indexes. True memory operates at a higher layer: the continuous negotiation between storage, relevance, temporal decay, and reasoning. Retrieval and windowing provide mechanical scaffolds that surface facts. Memory, properly conceived, drives dynamics. What deserves retention? What should expire? How does an assistant reconcile contradictions across time? When does a pattern carry more weight than an isolated fact? These challenges extend beyond clever engineering; they demand systems that reason about the validity of information as it shifts. A next-gen memory system advances beyond scale & search. It develops temporal awareness, validates knowledge against reality, and forgets with as much intention as it remembers. Assistants that only archive facts remain brittle. Assistants that balance storage, decay, and reasoning evolve into adaptive partners capable of growing alongside the world they inhabit. Read my latest take for a deeper dive into how I see the memory layer shaping: https://lnkd.in/e_g9am39

32

3 Comments

📑 New Survey: The Landscape of Agentic Reinforcement Learning for LLMs A comprehensive survey has just been released, synthesizing 500+ recent works on how reinforcement learning is transforming large language models from passive text generators into agentic decision-makers. 🔑 Key highlights from the paper: - Formalizes the shift from LLM-RL (single-step MDPs) to Agentic RL (long-horizon POMDPs) - Proposes a twofold taxonomy: By capabilities: planning, reasoning, tool use, memory, self-improvement, perception By applications: code agents, math agents, GUI agents, vision/embodied agents, multi-agent systems, and more - Consolidates open-source environments, frameworks, and benchmarks into a practical compendium for researchers - Discusses open challenges: trustworthiness, scaling agentic training, and scaling environments

113

4 Comments

Having worked on Agentic systems and open source multi agent deployments for last 2.5+ years (Starting with Langchain and moving to Langgraph, Autogen, Google ADK and many others) two common theme that has come across from deploying Agentic solutions and building using Agentic ecosystems in large scale production systems are - Theme 1 -> Agentic AI is shifting from experiment to enterprise backbone—but its ROI hinges on mastering cost containment and correctness at scale. In multi‑agent ecosystems, uncontrolled reasoning calls lead to spiraling spend with unpredictable accuracy (hence its very important to balance the choice of models, fault tolerant modes and accuracy metrics being defined both at Agent, Tool and Orchestrator levels). The winning CIO strategy is layered orchestration: lightweight models for routing, specialized LLMs for high‑value reasoning, and evaluator agents for correctness. This approach converts raw compute consumption into governed, auditable AI “throughput,” reshaping AI from an R&D cost center into a measurable productivity engine. Theme 2- > Standards like Google’s ADK and the Agent‑to‑Agent (A2A) protocol now provide the backbone for this transformation. By enabling a common language for agent communication, context‑sharing, and validation, they allow enterprises to modularize their AI stacks—aligning model choice with both business logic and budget. The value add: reduced infrastructure burn, accelerated time‑to‑market, and AI systems with built‑in accountability. For CIOs, this means a direct path to higher ROI curves where innovation can also take into account the true business value and cost of running these multi-agentic systems in production. #AgentAI #AgentOrchestration #CIOInsights #AIfrastructure #ADK #A2AProtocol

171

12 Comments

Why Kimi K2 is the most important AI Model Released this year Not because of its eval scores — but because it quietly rewrites the blueprint. While others scale FLOPs, Kimi scales structure. The team at Moonshot built something different: → A second-order-inspired optimizer (Muon) that converges faster by respecting geometry — stabilized with a per-head clipping mechanism that prevents divergence mid-training. → A fully automated skill factory: rubrics, tool simulators, LLM-based verifiers — all stitched together to manufacture agentic reasoning at scale. → A recursive alignment loop: critics are trained, tested, and continuously re-calibrated against objective truth before they’re allowed to judge subjective quality. This isn’t a model. It’s a protocol. And it signals the end of the monolithic model era. The next platform wars will be won by systems that: Generate, verify, and adapt in feedback loops, Make behavior programmable, and hold alignment under pressure Full breakdown here below. For original formatting see this- https://lnkd.in/dyQhWPBh

29

2 Comments

With the shift from one-shot answers to complex reasoning, the demand for inference is exploding. With that, AI is moving from pilots to AI factories — infrastructure that manufactures intelligence by turning data into tokens and decisions in real time. Decisions for every aspect of business, top-line, bottom-line, and people. The NVIDIA Blackwell platform delivers unmatched performance and best overall efficiency for AI factories. #NVIDIA #AI

42