Karthick Iyer

𝗠𝗶𝘅𝘁𝘂𝗿𝗲 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝘁𝘀 (𝗠𝗼𝗘): 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗟𝗠𝘀 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗹𝘆 𝘄𝗶𝘁𝗵 𝗦𝗽𝗮𝗿𝘀𝗲 𝗖𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻 Large Language Models (LLMs) continue to grow in size, pushing the limits of AI capabilities but also introducing challenges in cost, memory, and inference speed. Mixture of Experts (MoE) offers an innovative approach by using sparse computation, activating only a subset of parameters per input. Let's explore recent advances in MoE architectures and how models like DeepSeek-v2 and DeepSeek-v3 are optimizing efficiency. 🔹 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 𝗶𝗻 𝗠𝗼𝗘: 𝗥𝗼𝘂𝘁𝗶𝗻𝗴 𝗕𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸𝘀 & 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝗿𝗮𝗱𝗲-𝗼𝗳𝗳𝘀 While MoE improves efficiency, it also faces key challenges: 𝗧𝗼𝗸𝗲𝗻 𝗗𝗿𝗼𝗽𝗽𝗶𝗻𝗴 𝗶𝗻 𝗟𝗼𝗻𝗴 𝗦𝗲𝗾𝘂𝗲𝗻𝗰𝗲𝘀: OpenMoE struggles with routing stability, sometimes losing tokens in long sequences. Fixed Routing in Pretraining: Early routing patterns can be inefficient post-training. 𝗗𝗼𝗺𝗮𝗶𝗻 𝗦𝗵𝗶𝗳𝘁 𝗜𝘀𝘀𝘂𝗲𝘀: MoE models may struggle to generalize across different data distributions. A recommended solution is incorporating instruction-following data in pretraining to enhance routing adaptability. 🚀 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗠𝗼𝗘: Smarter Scaling for AI Models The DeepSeek series addresses these issues with innovative optimizations: 🔸 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝘃𝟮: 𝟮𝟯𝟲𝗕 𝗣𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀, 𝟮𝟭𝗕 𝗔𝗰𝘁𝗶𝘃𝗲 1️⃣ Multi-Head Latent Attention (MLA): Cuts memory use by 93% with efficient KV cache storage. 2️⃣ Fine-Grained Expert Allocation: Balances shared and specialized experts across devices. 3️⃣ Device-Level Load Balancing Loss: Ensures even routing across devices, improving stability. 🔸 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝘃𝟯: 𝗔 𝟲𝟳𝟭𝗕 𝗣𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿 𝗠𝗼𝗱𝗲𝗹 𝘄𝗶𝘁𝗵 𝗡𝗲𝘄 𝗘𝗻𝗵𝗮𝗻𝗰𝗲𝗺𝗲𝗻𝘁𝘀 1️⃣ Multi-Token Prediction (MTP): Predicts multiple tokens at once for better efficiency. 2️⃣ Auxiliary-Loss-Free Load Balancing: Dynamically adjusts expert selection without added inefficiencies. 3️⃣ FP8 Mixed Precision Training: Reduces training costs significantly (~$5.6M for full training). 4️⃣ Extensive Post-Training: Includes context extension (128K tokens), SFT, RLHF, and knowledge distillation. 📊 𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀 ✅ Trained with 2.78M H800 GPU hours ✅ Performance rivals top closed-source LLMs ✅ Practical, scalable MoE for real-world deployment 🔮 𝗧𝗵𝗲 𝗙𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗠𝗼𝗘: 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗔𝗜 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 MoE is revolutionizing LLM training, making sparse computation viable at scale. While early MoE models had challenges, recent breakthroughs like MLA, MTP, and smarter load balancing are proving MoE's potential. DeepSeek-v3 shows that sparse models can match dense models, signaling a shift in AI scaling strategies. What’s your take on MoE architectures? Will they define the future of AI, or do dense models still have an edge? Let’s discuss! 👇 credit : Cameron R. Wolfe, Ph.D.

185

20 Comments

As part of the IndiaAI Mission, Avataar is building UAI (Unified AI Interface) - a foundational DPI layer with thin, durable rails for trust and interoperability, connecting models, data and compute into a cohesive, discoverable ecosystem. We are privileged to collaborate with Prof. Ramesh Raskar (MIT) and Project NANDA: Architecting the Internet of AI Agents in shaping a broader “Internet of AI Agents” vision - exploring how a discovery layer for India’s evolving digital public infrastructure can unlock openness, innovation and responsible scale together. Equally grateful to Aafreen Siddiqui Sherwani (United Nations) for anchoring the dialogue, and to Pradeep Gupta (NVIDIA), Tanvi Lall (EkStep Foundation), Vineet Singh (Digital Green) and Jibu Elias (Mozilla Foundation) - for a thoughtful, forward-looking exchange on what it truly takes to build open AI platforms for India. The real opportunity ahead is not just technical, it is architectural. How we design these rails now will shape how intelligence compounds across citizens, enterprises and governance systems for decades. What principles should guide the design of AI infrastructure at national scale? #IndiaAI #DigitalPublicInfrastructure #ViksitBharat

161

9 Comments

This week, we announced that Qualcomm intends to invest up to $150 million through Qualcomm Ventures to support India‑based startups focused on AI for automotive, IoT, robotics, and mobile.   As AI enters a new phase where intelligence is being built directly into the devices and systems people rely on every day, India’s deep engineering talent and rapidly expanding startup ecosystem position founders to build scalable, real‑world edge AI solutions. https://lnkd.in/g2yENzQS

474

7 Comments

Samsung Semiconductor India Research is transforming chip design with Cadence Cerebrus AI Studio, achieving a PPA improvement of 8% to 11% on SoC subsystems. By utilizing features like Smart Model Replay and advanced data analytics, they've enhanced productivity and streamlined workflows. Discover the future of efficient chip design! https://ow.ly/RyNa30sMrSy

41

Blackwell’s unmatched performance and efficiency leadership comes from extreme hardware-software co-design. It’s a full-stack architecture built for speed, efficiency and scale including: NVFP4 low-precision format for efficiency without loss of accuracy Fifth-generation NVIDIA NVLink that connects 72 Blackwell GPUs to act as one giant GPU NVLink Switch, which enables high concurrency through advanced tensor, expert and data parallel attention algorithms Annual hardware cadence plus continuous software optimization — NVIDIA has more than doubled Blackwell performance since launch using software alone NVIDIA TensorRT-LLM, NVIDIA Dynamo, SGLang and vLLM open-source inference frameworks optimized for peak performance A massive ecosystem, with hundreds of millions of GPUs installed, 7 million CUDA developers and contributions to over 1,000 open-source projects

329

7 Comments

Cerebras Systems delivers the industry's fastest inference API. And our team of industry leading AI researchers advances the state of the art as well. The research described in this publication moves forward the rapidly evolving field of efficient ML inference. Cerebras Systems is committed to solving fundamental challenges in large-scale model acceleration. More detail: MASSV addresses fundamental challenges in multimodal speculative decoding through a novel two-phase approach. By grafting the target VLM’s frozen vision encoder onto a smaller language model and applying self-distilled visual instruction tuning, MASSV achieves remarkable improvements in token acceptance rates. Our technique significantly outperforms previous methods, increasing mean accepted lengths by up to 30% across multiple model families including Qwen2.5-VL and Gemma3.  https://lnkd.in/gcfZv9fS

158

3 Comments

As India’s R&D hubs scale at an unprecedented pace, we’re seeing a paradox unfold: greater innovation accompanied by rising burnout. In high-growth settings, it’s easy to get caught in the rhythm of rapid output and short-term wins. But the real breakthroughs in semiconductors, IoT, and wireless connectivity demand focus, patience, and long-view thinking. At Silicon Labs, we’ve institutionalized Quiet Weeks: dedicated company-wide time to minimize distractions, recalibrate, and engage in meaningful problem-solving. It’s not a perk, it’s a principle. One that acknowledges the cognitive rigor of engineering and fosters sustained innovation. This shift makes space for deeper thinking, more intentional collaboration, and gives early-career engineers the freedom to approach challenges expansively, not just reactively. Because in the race to build the future, it’s not about moving the fastest, it’s about thinking the deepest. #InnovationIndia #TechLeadership #LifeAtSiliconLabs

138

6 Comments

*** What does PAM4 brings to the High-Speed Designs *** PAM4 (Pulse Amplitude Modulation with 4 Levels) brings several critical benefits—and some challenges—to high-speed digital designs, especially in domains like SerDes links, high-speed memory (like HBM3), PCIe 6.0+, 800GB Ethernet, and chiplet interconnects (e.g., UCIe, CXL). Key Benefits of PAM4: o Doubles Data Rate Without Doubling Frequency.  - Encodes 2 bits per symbol (4 levels), allowing 56G/112G SerDes and PCIe 6.0 links at lower clock speeds. o Better Utilization of PCB and Interconnects  - Reduces required channel frequency, minimizing signal attenuation and easing material constraints. o Enables High-Density Chiplet Interconnects  - Used in UCIe and CXL to transmit more data over limited physical pads and bumps. Challenges of PAM4 and Mitigations: o Lower SNR / Smaller Eye Openings  - Use equalization, CTLE, and FEC. o Higher Sensitivity to Jitter - Employ advanced CDR techniques. o Increased SerDes Complexity  - Requires precision ADC/DACs and adaptive calibration. What is your experience in designs using PAM4?

14

AI is accelerating faster than any technology our industry has seen . . . and it’s transforming chip design from the inside out. At Synopsys Inc, we’re pushing the boundaries of what’s possible with generative and agentic AI, partnering closely with innovators like Microsoft and NVIDIA to bring new levels of automation, intelligence, and productivity to engineering teams.   From AI assistants that help engineers move faster to emerging AI “colleagues” that reason, plan, and take action across complex design flows, the future of semiconductor development is changing . . . fast!   I talk more about this in my recent blog: Generative and Agentic AI, explained. Check it out: https://lnkd.in/gmW36RxA

161

2 Comments

Proud to Partner with the University of Illinois Urbana-Champaign on Heterogeneous Computing! At AMD we are thrilled to collaborate with the University of Illinois Urbana-Champaign on the Heterogeneous Accelerated Compute Cluster (HACC) program. This innovative program, integrates CPUs, GPUs, and FPGAs, paving the way for groundbreaking research in heterogeneous computing. Why is this collaboration crucial? Addressing Complex Technical Challenges: HACC enables exploration of system interactions, tackling real-world problems with advanced computing solutions. Strengthening Academic-Industry Ties: Our partnership ensures that researchers have the resources and feedback needed to drive impactful research. Championing Open Source: The success of this collaboration is the open-source projects that receives contributions from more than 60 active members and highlights the importance of accessible technology. At AMD, we are dedicated to making AI solutions more accessible and user-friendly, empowering everyone to leverage our advanced computing resources to innovate and make a difference. Join us in celebrating these advancements and the collaborative efforts that make them possible! 🔗 Watch Now https://lnkd.in/g2V9XBV3 #HeterogeneousComputing #Innovation #Research #Education #AMD #OpenEcosystem #TechCollaboration #AI

101

3 Comments

"By combining 5G infrastructure and AI acceleration in a single, programmable chip, [EdgeQ] is not only challenging global incumbents but also establishing India as a hub for advanced wireless and AI semiconductor design." - Yashasvini Razdan See the full article coverage of EdgeQ Inc.: "How EdgeQ Is Building India’s First Unified 5G + AI SoC" https://lnkd.in/gSNWNqgP

128

1 Comment

How does a startup accelerate development with limited time and resources? At Akeana, co-founder Nitin Rajmohan reveals how Cadence's like Xcelium, Verisium Debug, and Palladium platforms helped them go from simulation to emulation seamlessly, speeding up their development cycle and delivering high-quality IP solutions. Read more: https://ow.ly/20nE30sP3aO

7

Ensuring reliable silicon, especially at advanced nodes, needs a methodology that spans the entire silicon life cycle - from design to manufacturing to mission mode usage. The post below from Shankar Krishnamoorthy and Yervant Zorian provides a comprehensive overview of what design teams can do to mitigate and tackle problems such as SILENT DATA CORRUPTION, a problem that becomes infinitely more prominent due to training demands of AI and GenAI. Other interesting links: From Meta: https://lnkd.in/gyeh6f9j From SemiEngineering: https://lnkd.in/g7pFwSVs https://lnkd.in/g6RQ_tX7 #synopsys #reliability #FuSa #3DIC

41

Spoiler: semiconductors are about to get a plot twist (and that’s a good thing). Loved sharing the startup perspective at #AIInfraSummit on how AI is bending the rules of chip design with Synopsys Inc and Silicon Catalyst. Some of my takeaways: 1) #AI is reshaping the way we think about #chipdesign; soon, intelligent agents and physics-informed foundation models will be co-designing with us 2) Data is at the core of this transformation and will lead to super-charging #semiconductor partnerships and ecosystems 3) Are semiconductor startups at a disadvantage when it comes to AI? Established companies certainly have valuable data-sets but AI is lowering barriers to entry and democratizing chip design at an astonishing rate. A big THANK YOU to our awesome moderator Sally Ward-Foxton at EE Times | Electronic Engineering Times and the many of you who joined the conversation — from creative questions during the session to great discussions after the panel Learn more about Cognichip and our Artificial Chip Intelligence - ACI(R) technology at https://www.cognichip.ai. Global Semiconductor Alliance Semiconductor Industry Association

70

2 Comments

#CES feels different this year. The inflection of AI moving from the cloud to the physical world is happening now. For AMD Physical AI is front and center this week. Physical AI breaks out of screens and into physical systems that sense, decide, and act in real time, under real world constraints. This is one of the hardest challenges in computing – where downtime is unacceptable, latency matters, and safety is non-negotiable. For AMD, Physical AI isn’t entirely new territory. We’ve been enabling this kind of real-time, mission-critical intelligence for more than two decades. Our technology already runs factory robots at micron-level precision, powers AI that inspects infrastructure as it’s being built, and supports surgical robotics where accuracy and reliability directly impact outcomes. Physical AI may be a new label, but we’ve had the foundation in place for years. At CES, we’re showcasing how the full AMD embedded portfolio comes together to enable that future – from healthcare and automotive to industrial systems and humanoid robotics. When AI has to interact with the real world, performance per watt, determinism, and system integration are what separate demos from deployable systems. That’s why I’m excited to share that today we launched the AMD Ryzen AI Embedded Processors. Built on “Zen 5,” with RDNA 3.5 graphics and the XDNA 2 NPU, these processors bring CPU, GPU, and NPU together in a single chip to deliver low-latency, energy-efficient AI for automotive cockpits, industrial automation, and autonomous physical systems. This week you’ll see it live: A humanoid concept robot processing touch, force, and vision data on AMD embedded processors Embedded AI driving real-time perception, analytics, and decision-making Automotive and industrial demos focused on safety, determinism, and long lifecycle deployment Using the breadth and depth of the AMD portfolio, we’re building systems that work reliably in the real world. We’ve laid the foundation and at CES, we’re showing what’s possible. If you’re in Vegas, come check out our booth. #PhysicalAI #Embedded #Innovation #TogetherWeAdvance

409

9 Comments