Carter Huffman

Giving a talk on "AI Evals – Rethinking Correctness for Programs When LLMs Are Involved" at MCCIA, Pune, on 30th Nov, at the OAISYS AI Practitioners Conference. Here's the abstract: If you incorporate Gen AI/LLMs in a software product, writing the program is the easy part. Ensuring that the program consistently produces correct results becomes much harder. Checking whether a program "works" becomes a much more difficult question to answer, because the answer is probabilistic, and that's because the program's output is probabilistic. And even if a program works when you release it, ensuring that it continues to work in production is also a difficult problem because the error rate can change, because inputs are open-ended, and edge cases emerge from model quirks rather than code paths. The session walks through the fundamentals of the emerging field of “Evals” — from simple keyword and pattern-matching methods to advanced approaches like LLM-as-a-Judge. Navin also discusses how to capture test cases, document unpredictable behavior, and balance accuracy with cost and effort when evaluating multi-turn conversations and AI-driven interfaces.

31

1 Comment

Researchers present NeuroSPICE, a physics-informed neural network framework for device and circuit simulation, offering unique advantages in design optimization and inverse problems. https://lnkd.in/gRYYqxQB

18

Nvidia Ising dropped yesterday, a 35B-parameter vision-language model that automates quantum processor calibration. It looks at calibration plots (T1 decay, readout discrimination, Rabi oscillations) and classifies them as SUCCESS or FAILURE. It also ships with a Decoder CNN for real-time quantum error correction. Both are open-source. It's genuinely impressive. Calibrating a 156-qubit chip by hand takes hours. Ising does it in minutes. It outperforms Gemini, Claude, and GPT on Nvidia's new QCalEval benchmark. And it plugs into agentic workflows. The VLM can drive a calibration loop that tunes the processor until error rates drop below threshold. So I pointed it at IBM's ibm_fez (156 qubits, 352 CZ coupling edges) and layered a Nervous Machine causal analysis on top to see what each tool finds. The result: 8 qubits that Ising would confidently label as healthy are completely unusable for any 2-qubit circuit. These qubits have excellent individual metrics, T1 coherence times above 110μs, readout errors below 3.4%. If you showed the VLM their calibration plots one at a time, it would say SUCCESS every time. Each qubit, in isolation, is fine. But their CZ gates are dead. 100% error rate with their neighbors. Any quantum circuit compiled to use these qubit pairs will fail with absolute certainty, despite every individual qubit in the pair appearing healthy. Why does the VLM miss this? It processes one calibration image per call. One experiment, one classification. It has no coupling map, no neighbor data, no cross-qubit correlation. When it looks at Q71's T1 decay curve and sees 130.8μs, it correctly says SUCCESS. It has no mechanism to know that Q72 next door is dead and that the CZ gate between them has a 100% error rate. This isn't a bug in Ising. It's a structural limitation of per-experiment classification. NM operates as a causal layer across the full chip, 156 qubits × 7 days × 352 CZ edges in a single pass. Instead of classifying one experiment at a time, it correlates gate errors across the coupling map and tracks certainty on every causal edge independently. NM found 4 failure clusters: Cluster 1 (Q27, Q28): Both qubits healthy individually. CZ gate between them: 100% error. Likely shared control line failure. Cluster 2 (Q32, Q33): Same pattern. Shared control line. Cluster 3 (Q71, Q72, Q73): Q72 is dead (T1=11.4μs, readout error 48.8%). Its death poisons the CZ gates to both neighbors. Q71 and Q73 look healthy but can't do 2-qubit operations. Cluster 4 (Q102, Q103): Same pattern as Clusters 1 and 2. Together: Ising classifies individual experiments fast. NM correlates those classifications across the chip to detect infrastructure-level failures. When NM detects a cluster, it can direct Ising to re-examine specific qubits with targeted prompts. When Ising flags an unexpected failure, NM can determine whether it's isolated or part of a pattern.

24

4 Comments

Jiakun Fan's upcoming ACM International Conference on Supercomputing 2026 paper on diffusion model serving is of course another good reason to visit Belfast this summer and attend ICS. While you are contemplating your travel plans and flight options, take a look at the code and give it a go, it's deployment-ready! Paper pre-print: https://lnkd.in/es6NTJ2D Code: https://lnkd.in/e4YP5FDn #VTCS #SustainableAI #ICS

14

Voice chat is one of the most powerful tools for building multiplayer communities, and one of the toughest to moderate at scale. Recently, a common refrain in the industry has been: “Just use an LLM.” But effective voice moderation takes far more than language modeling. At #GDC2026 today, based on our work at Modulate, Rachel Manzelli and I are digging into what it actually takes to build real-world voice moderation systems, and where LLM-only approaches fall short. If you’re around today, join us for our fireside chat. Would love to connect!

32

I've been exploring GCC's experimental C++26 compile-time reflection support (based on P2996) and built a small repo of working examples. These include: • Python style printing/formatting for structs and aggregates without manual boilerplate. • Compile-time optimization of struct layout to reduce padding • Enum to string compile-time conversion. • Serialization/deserialization of aggregate types using reflection. Echoing Herb Sutter’s view, I think reflection will be a game-changer for modern C++, similar to the impact constexpr had. Repo: https://lnkd.in/gPSADQjR

75

I’m excited to share a new extension I built for GitHub Spec Kit: Spec Kit Retrospective! If you’re exploring Spec-Driven Development (SDD) with AI agents, you’ve probably run into this exact issue: you go through /specify, /plan, /tasks, and /implement, but the final codebase inevitably diverges from your original spec. What happens to that lost context? Usually, the specs become outdated, and the AI agent repeats the same mistakes on the next feature. Or, even better, when the agent ends the task, It doesn't update the specs! I created Spec Kit Retrospective to close that feedback loop. This extension adds a crucial final phase to the workflow, allowing your AI agent to: - Analyze the gap between the initial plan and the final implemented code. - Capture learnings and automatically re-adjust specs after divergence. - Feed insights back into your project's memory and constitution to mitigate future spec-drift. Instead of losing the "why" behind those last-minute code changes, this tool helps your AI learn from the implementation phase so your next feature build is even sharper. Check out the repository, feel free to give it a try (or a star!), and I’d love to hear your feedback! https://lnkd.in/epYxu6Hz

10

This one is wild. So proud to present TAC: Timestamped Audio Captioning, a project lead by Sonal Kumar and Prem Seetharaman here at Adobe. TAC doesn’t just describe sounds: it _grounds_ them in time. Instead of a generic caption, you get something like: “Metallic sword scrape from 5s to 10s.” We extend this to video by cascading TAC with a VLM, producing timestamped audiovisual captions. You basically get automatic Closed Captions/Subtitles. The demos are honestly epic, definitely check them out! The core idea is simple: ground first, reason second. By decomposing the problem this way, we achieve SOTA results across several audio and audiovisual understanding benchmarks. What a titanic project with a titanic team! Check out the paper and demo here: https://lnkd.in/gfdspYjD

44

2 Comments

There is a new robotic revolution in terms of hardware! Yet, robotics software stagnated at ROS for some reason. While ROS is what people are using, it presents a significant hurdle for many of us in the ML field to get started (interfacing with the hardware, C/C++ 🤯 ). Well, we want to change that! Welcome to the new generation of robotics software, based on Python! Yep, Python! So you can pip install robotics!! The paper is coming to arxiv, but as usual, my LinkedIn followers get a first view first, so have a read through the attached paper! We are opening it step by step to allow feedback and engagement, please star our repos: https://lnkd.in/eBDFmuwu Website: https://lnkd.in/eTcNiTRG 🚀 Meet Ark – a Python-first, open-source framework that finally lets ML engineers and roboticists speak the same language. One pip install, no labyrinthine C++ build chains. 🔄 One flag = sim-to-real. Toggle sim: True/False in a YAML file and run the exact same policy on PyBullet/MuJoCo or physical hardware – no code rewrites, no ROS launch-file plumbing. 🧩 Node-based, LCM pub-sub architecture means sensors, policies, and controllers are hot-swappable processes. Hack, crash, or iterate without taking the whole stack down. 📦 Built-in data & debugging tools – LCM logger/player, real-time plots, graph viewer, and camera streams – slash time spent on “why is my topic empty?” detective work. 🏋️ Out-of-the-box imitation-learning pipelines (Diffusion Policy, ACT) with reusable data-collection nodes (VR, gamepad, kinesthetic). RL integration is next on the roadmap. 🤝 Interoperates, doesn’t dictate – optional ROS bridge, clean C/C++ bindings for real-time loops, and a backend API that can host any simulator or custom driver. 🗺️ SLAM → Planning → Control modules included: Fast-SLAM mapping, A* path planning with safety margins, PD waypoint tracking – demoed on a Husky in a kitchen maze. 🤖 Case studies: ViperX pick-&-place, A1 humanoid cloth folding, language-conditioned board-game play via DeepSeek-R1, mobile inspection – all reproducible from provided configs. 👉 Call to action: Check out the repo, try Ark on your lab robot this week, and let us know what you build! With a fabulous team: Magnus Dierking Christopher Mower Sarthak Das Henlog, Jiacheng Qiu Cody Reading Wei C. Huidong Liang Huang Guowei Jan Peters Quan Xingyue Jun Wang

172

12 Comments

4th publication and my 1st journal paper in IEEE Access. Truly grateful to Prof Mark Lee Sir and Prof Aruna Malapati Ma'am for their constant guidance, mentorship, and support throughout this journey. This work introduces MDMIC, a benchmark dataset designed to address complex cross-domain, multi-intent NLU challenges in six low-resource Indic languages. Our unified multilingual, multitask architecture with attention-based fusion demonstrates consistent performance gains across intent detection, domain classification, and slot filling, advancing robust conversational AI for low-resource settings. Link: https://lnkd.in/ga27aFFC

18

2 Comments

https://lnkd.in/gMv6U_Mt : RLVR was the all the rage with deepseek and alphaproof becoming expert coders and mathematicians. Turns out they are "smart workers" and not "scientists". If base model can answer, RL can make it faster and cheaper 90% of the time. If base model cannot, RL just can't. Read more about it in nios!

15

Our paper, Spherical ViTs for Audio-Visual Saliency Prediction in 360° Videos, has been accepted to IEEE TPAMI. This work extends our BMVC’23 work by incorporating spatial audio, enabling more accurate saliency prediction in VR/360° environments. To tackle the lack of audio-visual benchmarks, we introduce YT360-EyeTracking: a collection of 81 360° videos with eye-tracking data recorded under three audio conditions—mute, mono, and spatial (ambisonic). This is the largest and most systematic dataset available for studying audio-visual saliency in VR. This work is the outcome of a great collaboration with my friend Halit Ozsoy and dearest supervisors Dr. Nevrez Imamoglu, Cagri Ozcinar, Assoc. Professors İnci Ayhan, Aykut Erdem, and Professor Erkut Erdem. You can find the code, dataset, and qualitative results on our project page: https://lnkd.in/dewh5fS5. https://lnkd.in/d7fTjUgF Preprint: arxiv.org/abs/2508.20221

64

Code retrieval is one of the biggest bottlenecks in agentic coding. When agents work on large codebases, up to 60% of time is spent searching for relevant files.   Cognition approached the problem differently.   Their SWE-grep models - compact, RL-tuned systems built for deep code retrieval - run on Cerebras. Developers can now pull relevant code from massive repositories - millions of lines - in seconds, not minutes.   End-to-end workflows in Windsurf run up to 5× faster than Anthropic's Sonnet When scaling laws were discovered, the world fixated on size - bigger models, bigger clusters.   But if inference is fast enough, you can call smaller, specialized models many times - and outperform the giants.   That’s the world we’re entering now. This is what happens when fast inference meets thoughtful model design.   Congratulations to the Cognition team on an impressive ML breakthrough. We’re proud to put our wafers to work for every Windsurf user. Cerebras Systems + Sonnet on the left. Sonnet alone on the right.

250

8 Comments

Check out this podcast: Bridging Circuits and Concepts in Large Language Models https://lnkd.in/gmDDHj5m How do millions of computations inside large language models add up to something like understanding? This episode explores the latest breakthroughs in mechanistic interpretability, showing how tools like representational geometry, circuit decomposition, and compression theory illuminate the missing middle between circuits and meaning.

3

#TuesdayPaperThoughts Edition 55: AggLM - Learning Solution Aggregation This week's #TuesdayPaperThoughts explores "The Majority is not always right: RL training for solution aggregation" from FAIR at Meta. The paper challenges the standard practice of majority voting in test-time compute scaling by learning aggregation as an explicit reasoning skill. Key Takeaways: 1️⃣ Beyond Majority Voting: AggLM treats solution aggregation as a learnable skill rather than a fixed heuristic. Given multiple candidate solutions, the system trains an aggregator model to review, reconcile, and synthesize a final answer using reinforcement learning from verifiable rewards. This enables recovery of minority-but-correct answers that majority voting would miss. 2️⃣ Balanced Training Strategy: The training methodology carefully balances easy examples (where majority is correct) with hard examples (where majority is wrong). This dual approach allows the model to learn both selection of correct majorities and synthesis of minority solutions, with the optimal mixture being around 50% easy examples. 3️⃣ Superior Performance & Efficiency: AggLM-1.7B achieves 50% accuracy on AIME25 compared to 45% for majority voting, while using substantially fewer tokens. The method generalizes effectively across different solution models, including stronger ones not seen during training, and scales better with additional candidate solutions. We loved that the approach can transform aggregation from a simple counting mechanism into sophisticated reasoning. Research Credits: Wenting Zhao, Pranjal Aggarwal, Swarnadeep Saha, Asli Celikyilmaz, Jason Weston, Ilia Kulikov Paper Link: In comments

19

1 Comment