Hatem Elseidy

Anthropic used 16 AI agents to build a C compiler from scratch in two weeks for about $20,000. The result was 100,000 lines of code that compile the Linux kernel and pass 99% of GCC’s torture tests, while also compiling FFmpeg, Redis, PostgreSQL, QEMU, and even running Doom. For comparison, GCC took thousands of engineers and 37 years to develop.

149

10 Comments

If you're interested in PyTorch internals, the PyTorch Developer Podcast is a great resource! Each episode dives into internal dev topics in bite-sized (10-20 min) discussions. Recent episodes cover: ►  Compiler collectives – How PT2 compiler instances communicate via NCCL ►  TORCH_TRACE & tlparse – Logging & parsing for PyTorch 2 compilation ►  Higher-order operators – More flexible torch.ops with callable arguments ►  Inductor post-grad FX passes – Optimizations after functionalization Link in comments

24

🚀 The Simplicity of Sound: A Leaner Approach to Audio Compression Recent advancements in neural audio codecs have largely focused on superior audio quality via increasingly intricate architectures, often using multiple quantizers for encoding. But in the pursuit of fidelity, have we overlooked the practical limitations imposed by the sheer computational cost and complexity of these multi-stage quantization processes? This research introduces SQCodec, a novel neural audio codec that embraces elegance—achieving high-fidelity audio compression through a significantly leaner architecture built around a single quantizer. Its innovation lies in a carefully engineered design featuring streamlined convolutional networks, local Transformer modules, and the introduction of TConv, a mechanism that efficiently captures both short- and long-term acoustic variations. By adopting a single quantizer and a lightweight structure, this design tackles the challenges of complex multi-quantizer systems, such as high computational and memory demands that hinder real-world scalability. It also simplifies the tokenisation process, avoiding hierarchical token streams that complicate integration with downstream generative models and require specialised aggregation. While using a single quantizer risks reduced codebook capacity, potentially impacting reconstruction accuracy, this is addressed via Finite Scalar Quantization (FSQ), which supports large codebooks without typical collapse issues. At very low bitrates, the Mel Spectrogram Distance (MEL) may be slightly higher than models optimized for it, as this codec prioritises STFT distance during training. The model was trained on diverse datasets—LibriSpeech, Common Voice, MTG-Jamendo, and FSD50K—proving its ability to generalize across domains. Training was completed efficiently on a single NVIDIA RTX 4090 GPU, showcasing its low resource demands. Crucially, SQCodec matches the audio quality of state-of-the-art multi-quantizer codecs like DAC while using far fewer parameters and significantly less computation. At ultra-low bitrates, it often outperforms other single-quantizer models like WavTokenizer across several objective metrics. This work paves the way for a future where neural audio compression is more accessible and integrable, particularly in resource-constrained scenarios and applications like real-time communication and multimodal LLMs. SQCodec's success underscores the potential of efficient audio processing and hardware-aware design. #LightweightAudio #SingleQuantizer #NeuralCodec #EfficientAI #AudioCompression

27

I finally came around to reading Meta AI’s “DINOv3” technical report, and it feels like a major milestone for self-supervised learning (SSL)! For a while, the vision foundation model space has seen a split. On one side, you have weakly-supervised models like CLIP, trained on image-text pairs, which excel at global, semantic tasks. On the other, you have SSL models like DINOv2, trained on raw pixels, which produce exceptionally strong dense features for geometric tasks. We all know SSL holds the promise of limitless scaling without annotations, but scaling has a notorious problem: dense features tend to fall apart during long training runs on massive models. What if you could solve this degradation and unlock the full potential of scaling SSL? The authors present a set of simple yet powerful strategies to successfully scale a DINO model to 7 billion parameters on a new 1.7 billion image dataset. The goal is to create a single, frozen backbone that can serve as a universal visual encoder. But the most interesting part isn't just the scale. It's how they fixed the feature degradation. They introduce a new method called Gram anchoring, which effectively addresses this known-but-unsolved issue. As training progresses, the patch-level features become "noisy" and lose their precise localization. Gram anchoring uses an earlier, "cleaner" version of the model as a teacher to regularize the structure of similarities between patches (the Gram matrix), forcing the model to maintain local consistency without stifling feature evolution. So, how well does it work? The DINOv3 family of models establishes a new state-of-the-art on dense tasks, significantly outperforming all other self- and weakly-supervised models, including those that distill from supervised segmentation models like SAM. Insights from their approach: The paper clearly illustrates how, without intervention, performance on dense tasks like segmentation peaks early and then steadily declines as training continues, even as global classification improves. The feature maps literally become noisy and less useful. Gram Anchoring is the fix: the most compelling finding is how effectively this simple regularization "repairs" the corrupted dense features. Applying it late in training almost immediately restores and then surpasses the previous peak performance on dense tasks. Bottom line: It’s a powerful demonstration that the biggest hurdles in SSL are often subtle training dynamics, not just architectural or objective function design. The key takeaway is that by solving feature degradation, pure SSL can produce a single, universal visual encoder that sets a new standard for dense, geometric tasks while remaining competitive on global ones.

159

6 Comments

🚀 Gemma 3: Powerful, Efficient, and Ready to Run Anywhere Gemma 3 is a new collection of lightweight, state-of-the-art open models built for efficiency and performance. 🔹 Outperforms other models in its size class—ideal for single-GPU or TPU setups. 🔹 Supports 140+ languages with advanced text & visual reasoning. 🔹 128K-token context window with function calling for complex tasks. 🔹 Quantized versions for faster performance and lower compute costs. A step forward in making AI more accessible and efficient! https://lnkd.in/gUSMMQaU

10

🚀 Big news — NeMo Evaluator is now open source! 🧪 We’re thrilled to announce that NeMo Evaluator SDK — the engine behind NVIDIA’s internal model benchmarking — is now open sourced and available to everyone: 🔗 https://lnkd.in/e3yaznR4 NeMo Evaluator SDK is an open-source platform for robust, reproducible, and scalable evaluation of large language models. It lets you run hundreds of benchmarks across popular evaluation harnesses against any OpenAI-compatible model API, all from auditable Docker containers for trustworthy and reproducible results. NeMo Evaluator is built on four core principles: ✅ Reproducibility by Default: Every config, random seed, and dependency is captured automatically for repeatable results. ⚙️ Scale Anywhere: Run on your laptop, a Slurm cluster, or cloud backends like Lepton AI without changing your workflow. 📊 State-of-the-Art Benchmarking: Access 100+ benchmarks from 18 open-source harnesses, covering reasoning, math, code, safety, and more. 🧩 Extensible and Customizable: Add new benchmarks, integrate private datasets, or connect custom exporters to your existing MLOps tooling. This release is a major step toward transparent, standardized, and community-driven model evaluation. From here on, we’ll be building NeMo Evaluator in the open — and we’d love for you to join us: try it, break it, file issues, contribute, and help shape its future! ⭐️ Explore it here → https://lnkd.in/e3yaznR4 If you find it useful, don’t forget to star the repo and help us grow the community! #NVIDIA #NeMo #Evaluator #OpenSource #LLM #Evaluation #AIResearch #MLOps #Benchmarking

257

3 Comments

MISRAJ AI introduces Baseer (بصير), a specialized vision-language model that sets a new state-of-the-art for the notoriously difficult task of Arabic document Optical Character Recognition (OCR). By meticulously fine-tuning a general-purpose Multimodal Large Language Model (MLLM) on a massive, curated dataset, the researchers have created a model that not only extracts text but also understands document structure, significantly outperforming existing open-source and large-scale commercial systems. The creation of Baseer was a multi-stage process centered on data and an efficient training strategy: 1. Massive Hybrid Dataset Curation: The team built a 500,000-pair dataset specifically for Arabic document-to-markdown OCR. This dataset is a hybrid of: ✅ 300,000 Synthetic Documents: Created using an in-house pipeline to generate a vast diversity of layouts, fonts, and document types, including aggressive data augmentation to simulate real-world degradation. ✅ 200,000 Real-World Documents: Curated from a diverse collection of books, magazines, and academic papers to capture authentic layout complexities and typographic conventions. 2. Efficient Decoder-Only Fine-Tuning: The team chose the Qwen2.5-VL-3B-Instruct model as their foundation due to its strong baseline performance in Arabic. Crucially, they adopted a decoder-only fine-tuning strategy, where the pre-trained vision encoder was kept frozen. This preserved the model’s powerful, generalized visual features while allowing the language decoder to specialize entirely on the nuances of Arabic text and document structure. 3. A New, High-Quality Benchmark: To rigorously evaluate their work, the researchers also created and open-sourced Misraj-DocOCR, a new expert-verified benchmark for Arabic OCR, addressing significant errors and a lack of diversity in existing benchmarks. This is the key diagram from the paper, providing a clear and concise overview of the entire data and training pipeline, from the hybrid dataset to the final fine-tuned Baseer model. ➡  You can read the paper from here: https://lnkd.in/dhjaceCg ------------------------------ ➡ Do not miss the 50% discount on the To Data & Beyond yearly subscription: https://lnkd.in/dZ2Sjkhk ➡ All My 7 Books, One Button Away, With 40% Off: https://lnkd.in/dqzXcDyJ #llm #opensource #largeLanguagemodels

50

2 Comments

New version of the StripedHyena 2 paper is out on arXiv To learn about how we trained large (40 billion parameters) convolutional language models efficiently at one million sequence length, with custom context parallelism: 👇 All code is available paper: https://lnkd.in/ez7SmDwA code: https://lnkd.in/gHKTQJeR

116

1 Comment

Meta Introduces LlamaRL: A Scalable PyTorch-Based Reinforcement Learning RL Framework for Efficient LLM Training at Scale Meta researchers introduced LlamaRL, a fully asynchronous and distributed reinforcement learning framework. It is tailored for training massive LLMs on clusters ranging from a few to thousands of GPUs. They built LlamaRL entirely in PyTorch and implemented a single-controller design to simplify coordination. This design enables modular customization. Separate executors manage each RL component—such as the generator, trainer, and reward model—and operate in parallel. This asynchronous setup reduces waiting time throughout the RL pipeline. It also enables independent optimization of model parallelism and memory usage. LlamaRL’s architecture prioritizes flexible execution and efficient memory usage. It offloads generation processes to dedicated executors, allowing the trainer to focus exclusively on model updates. Distributed Direct Memory Access (DDMA) supports this offloading. It uses NVIDIA NVLink to synchronize weights in under two seconds—even for models with 405 billion parameters. The framework applies Asynchronous Importance-weighted Policy Optimization (AIPO) to correct for off-policyness caused by asynchronous execution. Each executor operates independently, leverages fine-grained parallelism, and applies quantization techniques to inference models to further reduce compute and memory demands...... Read full article: https://lnkd.in/gHn2QpSy Paper: https://lnkd.in/g5KBS_fq Meta AI at Meta

34

🚨 MLAAD v6 is out now! 🚨 We’re excited to announce the release of the 6th version of the Multi-Language Audio Anti-Spoofing Dataset (MLAAD). This update features some of the most recent and advanced text-to-speech (TTS) models, including: 🎙️ Dia TTS, 🎙️ Fish TTS, 🎙️ Spark TTS, 🎙️ Zonos TTS, 🎙️ Sesame CSM, 🎙️ Orpheus TTS, 🎙️ Mega TTS. We’re also proud to include samples from Resemble AI, a leader in speech synthesis solutions - further enhancing the dataset’s realism and diversity. 📊 MLAAD v6 stats: - 91 TTS models, - 42 distinct architectures, - 38 languages. This makes it one of the most comprehensive deepfake audio datasets available for advancing research in audio deepfake detection and voice anti-spoofing. 🔗 Dataset: https://lnkd.in/dYw_DQDb (now also on HuggingFace) 🔗 Paper: https://lnkd.in/dyAz2qfd A big thank you to all co-authors: Nicolas Müller, Wei Herng C., Edresson Casanova, Eren Gölge, Thorsten Müller, Piotr Syga, Philip Sperl and Konstantin Böttinger and Fraunhofer AISEC. #TTS #datasets #deepfakes #antispoofing #MLAAD #speechsynthesis #AI #opensource

84

2 Comments

We cut the hallucinations of an LLM in half using a new technique: RLFR (Reinforcement Learning from Feature Rewards). RLFR reduces hallucinations in Gemma 12B by 58% (when run with our probing harness), for ~90x cheaper per intervention than the LLM-as-judge alternative. The technique uses lightweight probes on a model's internal representations as reward signals for reinforcement learning. This is all while avoiding: - off-target effects (no degraded performance on standard benchmarks) - degrading/reward-hacking the probes (the probes still work as monitors at test time!) This is just our first public demonstration of our work building towards "intentional design": the ability to intelligently guide gradient descent, enabling a new paradigm of far more precise, robust, and effective training. Kudos to Aaditya Prasad, Connor Watts, Jack Merullo, Dhruvil Gala, Owen Lewis, Tom McGrath, and Ekdeep Singh for this groundbreaking new work!

211

9 Comments

#CVPR2025 Paper Picks #2 🛹RollingDepth Video depth estimation is hard: estimating depth per frame leads to flickering + temporal errors. 🎥 RollingDepth skips video diffusion—uses an image model on short frame snippets + global alignment to recover consistent depth over time. 💎 Works on 3-frame snippets with cross-frame attention, global depth alignment, and optional diffusion refinement — all from a single-image model. ▶️ Model: https://lnkd.in/dFAsGd8k ▶️ Demo: https://lnkd.in/dqPA-WB5 ▶️ Paper: https://lnkd.in/dWNP-Duf

98