Titus Winters

*** Cross-die links introduce skew, coherence issues, and latency that don’t appear in monolithic SoCs. *** Chapter 5 of my book on chiplets details die-to-die interconnects (UCIe, BoW, AIB, and others) and the hooks needed to validate them. Now available on Amazon in hardcover, paperback, and Kindle. Links in the comments.

19

1 Comment

By now, I imagine many of you have seen Giacomo Indiveri’s very well written, and intentionally provocative, article in Neuron: “Neuromorphic is Dead. Long Live Neuromorphic”. As a neuroscientist in neuromorphic reading a paper targeting the neuroscience community, I wanted to respond to some of the points raised. I can appreciate many of Giacomo’s concerns and I agree with much of it. That said, some of the arguments raised in the paper are similar to those that initially kept me away from the neuromorphic community, a field that I have since found to be incredibly rewarding and (largely) welcoming. Specifically the idea neuromorphic must be analog. Neuroscience (and science in general) stumbles when people start worshiping the model system rather than the goal. When we fail to look at complementary systems, or actively dismiss them, everyone suffers. In the case of neuromorphic, analog CMOS is just a model system, much like C Elegans. Personally, I have gained far more intuition about the brain from large scale digital platforms like TrueNorth, Loihi, and SpiNNaker, which the article is dismissive of because they aren’t what Carver Mead envisioned 40 years ago. I don’t personally get much benefit from small analog circuits, because the neuroscience questions I have asked focus on computing questions rather than physics questions. Giacomo is fair to say digital platforms abstract away continuous time and neuron complexity, but unlike analog CMOS, digital enables extreme scales, brain-like circuit complexity, and programmability.  When I look at the brain, I see an elegant efficient parallel computing system, and the old school neuromorphs see resistors and capacitors. And that’s okay – worms, flies, mice, rats, and monkeys all fail to capture important aspects of human brains yet we still study them. It allows more people to ask more questions from different scales. Analog circuit design shouldn't be a prerequisite for neuromorphic than worm optogenetics should be a prerequisite for neuroscience. The digital neuromorphic technologies criticized in his paper have brought comp sci, applied math, and systems neuro into the neuromorphic field. There is no way that is a bad thing. There is a lot I wish will change. I have long wished the memristor and AI fields would take neuroscience more seriously. I have worked with both fields and they often opt for near-term gains rather than looking to the brain. But we aren’t going to convince people looking at a trillion dollar industry by saying you’re either with us or you’re against us. One of the things that we have strived for at the NICE Conference is a big tent approach to neuromorphic computing and brain inspiration. Analog or digital, silicon or memristors, stochastic or deterministic. It doesn’t matter so long as the goal is to advance our understanding of the brain and the brain’s ability to perform computations that still elude us. Neuromorphic computing was never dead. It just grew up.

219

30 Comments

A neat perspective by my economist brother about the need for a new approach for publishing across interdisciplinary boundaries. Certainly a pain point for many emerging computing fields like neuromorphic where publishing in your department's familiar venues may conflict with reaching the broader community https://rdcu.be/ewD2D

17

1 Comment

Cerebras Systems inside of Cline provides lightning fast inference inside your developer workflow. Here is a quick tutorial showing just how easy it is. There are lots of things worth waiting for. Inference results aren't among them. Build with the fastest...

176

⭐ Excited to share our new blog post⭐ Intel Labs and Hugging Face Introduce enhanced Universal Assisted Generation (UAG) to 🤗 Transformers. Traditional speculative decoding requires the assistant and target models to share the same vocabulary. UAG mitigates this limitation by allowing the use of ANY target/assistant model pair, regardless of vocabulary differences 🤯. The latest UAG upgrade in 🤗 Transformers v4.50 now supports both greedy decoding and probabilistic decoding (sampling) for greater flexibility. See link to the blog post here: https://lnkd.in/dMBvcBk3 Thanks to the awesome collaborators: Jonathan Mamou,  Daniel Korat, Moshe Berchansky, João Gante, Lewis Tunstall, Nadav Timor, Moshe Wasserblat Roy Schwartz

80

1 Comment

Adding to the excitement around MPI 5.0: The MPI Forum's Hybrid and Accelerator Working Group has released the new MPI Memory Allocation Kinds side document! This document supplements MPI's Memory Allocation Info by defining new memory allocation kinds and their associated restrictors, enabling uniform support for diverse accelerator memories in MPI applications. Check out version 1.0 of the side doc and see how it can help streamline memory management in your MPI+accelerator programs. MPICH 4.3 already supports the memory allocation kinds API, and support in Open MPI 6.0.x is on the way. https://lnkd.in/gXsK-Weq

79

2 Comments

"Meh, who cares?" Firmware developers sometimes ignore thread safety and concurrency issues in their debug support code. "Meh, who cares?" one might think. In one project I reviewed, multiple global variables were being used to keep a count of certain events. The various count variables (three of them) were incremented in one concurrency context and decremented in another. The increment/decrements were certainly not being performed in a thread safe manner AND the logical grouping of the three global variables were also not concurrency safe. These variables were only used for certain debug print outputs. No big deal, right? Yes and no. Indeed, the product's primary function would never be impacted by my code review finding on this particular item. However, this firmware team had spent untold hours pouring over logs trying to find one elusive bug. What if the logs themselves could not be trusted? As with all concurrency issues, this was an easy mental trap to fall into. And trapped indeed was this team. Months of time consumed trying to find a single bug preventing a large order of their product. Logs were relied upon, yet the logs themselves were suspect. In the end, my review helped this team find the core issue, which was yet another concurrency bug. The team ignored my finding on these global debug variables. Time will tell if those variables lead the team astray with any future bug hunt. Still learning about concurrency? Take a look at this post of mine, which I believe covers this topic nicely from a firmware and embedded software point of view: https://lnkd.in/ecpBkjR

17

3 Comments

𝗠𝗶𝘅𝘁𝘂𝗿𝗲 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝘁𝘀 (𝗠𝗼𝗘): 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗟𝗠𝘀 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗹𝘆 𝘄𝗶𝘁𝗵 𝗦𝗽𝗮𝗿𝘀𝗲 𝗖𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻 Large Language Models (LLMs) continue to grow in size, pushing the limits of AI capabilities but also introducing challenges in cost, memory, and inference speed. Mixture of Experts (MoE) offers an innovative approach by using sparse computation, activating only a subset of parameters per input. Let's explore recent advances in MoE architectures and how models like DeepSeek-v2 and DeepSeek-v3 are optimizing efficiency. 🔹 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 𝗶𝗻 𝗠𝗼𝗘: 𝗥𝗼𝘂𝘁𝗶𝗻𝗴 𝗕𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸𝘀 & 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝗿𝗮𝗱𝗲-𝗼𝗳𝗳𝘀 While MoE improves efficiency, it also faces key challenges: 𝗧𝗼𝗸𝗲𝗻 𝗗𝗿𝗼𝗽𝗽𝗶𝗻𝗴 𝗶𝗻 𝗟𝗼𝗻𝗴 𝗦𝗲𝗾𝘂𝗲𝗻𝗰𝗲𝘀: OpenMoE struggles with routing stability, sometimes losing tokens in long sequences. Fixed Routing in Pretraining: Early routing patterns can be inefficient post-training. 𝗗𝗼𝗺𝗮𝗶𝗻 𝗦𝗵𝗶𝗳𝘁 𝗜𝘀𝘀𝘂𝗲𝘀: MoE models may struggle to generalize across different data distributions. A recommended solution is incorporating instruction-following data in pretraining to enhance routing adaptability. 🚀 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗠𝗼𝗘: Smarter Scaling for AI Models The DeepSeek series addresses these issues with innovative optimizations: 🔸 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝘃𝟮: 𝟮𝟯𝟲𝗕 𝗣𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀, 𝟮𝟭𝗕 𝗔𝗰𝘁𝗶𝘃𝗲 1️⃣ Multi-Head Latent Attention (MLA): Cuts memory use by 93% with efficient KV cache storage. 2️⃣ Fine-Grained Expert Allocation: Balances shared and specialized experts across devices. 3️⃣ Device-Level Load Balancing Loss: Ensures even routing across devices, improving stability. 🔸 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝘃𝟯: 𝗔 𝟲𝟳𝟭𝗕 𝗣𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿 𝗠𝗼𝗱𝗲𝗹 𝘄𝗶𝘁𝗵 𝗡𝗲𝘄 𝗘𝗻𝗵𝗮𝗻𝗰𝗲𝗺𝗲𝗻𝘁𝘀 1️⃣ Multi-Token Prediction (MTP): Predicts multiple tokens at once for better efficiency. 2️⃣ Auxiliary-Loss-Free Load Balancing: Dynamically adjusts expert selection without added inefficiencies. 3️⃣ FP8 Mixed Precision Training: Reduces training costs significantly (~$5.6M for full training). 4️⃣ Extensive Post-Training: Includes context extension (128K tokens), SFT, RLHF, and knowledge distillation. 📊 𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀 ✅ Trained with 2.78M H800 GPU hours ✅ Performance rivals top closed-source LLMs ✅ Practical, scalable MoE for real-world deployment 🔮 𝗧𝗵𝗲 𝗙𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗠𝗼𝗘: 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗔𝗜 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 MoE is revolutionizing LLM training, making sparse computation viable at scale. While early MoE models had challenges, recent breakthroughs like MLA, MTP, and smarter load balancing are proving MoE's potential. DeepSeek-v3 shows that sparse models can match dense models, signaling a shift in AI scaling strategies. What’s your take on MoE architectures? Will they define the future of AI, or do dense models still have an edge? Let’s discuss! 👇 credit : Cameron R. Wolfe, Ph.D.

185

20 Comments

10/25: Tenth in the series on 25 Google papers for 25 years of WSC: Google-Wide Profiling The "Always On" Monitor: Most companies turn profiling off in production because they fear the overhead. At Google, we keep it on. This paper (Google-Wide Profiling) explains how to get continuous, fleet-wide visibility with a tiny tax: just 0.01% overhead. Why bother, you might ask? Because "lab" environments can be misleading. By profiling in prod, you find: Hardware differences you didn't expect; "Cold" code that is actually running hot; Library bottlenecks across every binary in the fleet. It’s the difference between guessing where the performance went and _knowing_. And for today's nano-banana poster for the paper, I tried a different prompt: [Make an infographic academic-style poster for the following paper. Use a visual aesthetic similar to the architectural blueprint roll -- Large blue sheets with white technical lines, measurements, and room labels.]

377

12 Comments

On arXiv today, I posted a note https://lnkd.in/gxqWZB7t that describes how to build Josephson junction, inductor, and mutual inductance transformer models from basic library components in Keysight ADS. This is a follow up on our work to enable flux quantization in frequency domain circuit simulations https://lnkd.in/g5nkYNyD . Hopefully, this is detailed enough to allow users of other commercial or open-source circuit simulators to reproduce these results.

155

4 Comments