Caroline Liongosari

Exciting news! We have open-sourced the 7B sigmoid attention model (and its softmax baseline)! In Sigmoid attention, we replace the softmax operation with sigmoid (which is a lot faster). We confirm that with hybrid-norm, we can scale it up to 7B parameters and get equal performance vs the softmax baseline. Now you can get your hands on these models, and even their intermediate checkpoints. Both are trained with deterministic data, so at each intermediate step, they have seen the exact same data — perfect for further research. See our paper for the results, or directly run the notebook that shows how to do inference on the 7B models using AXLearn: https://lnkd.in/eEWYzC8U

287

9 Comments

✨ My Meta Interview Experience ✨ I recently had the opportunity to interview for the Data Scientist, Product Analytics role at Meta. While the outcome wasn’t what I hoped for, I walked away a better, sharper, and more resilient version of myself. Here’s a breakdown of my journey: 🔹 Initial Screen (45 mins): This was a fast-paced mix of SQL, product intuition, and metrics design. I had to break down a product problem, write clear, efficient code, and define success with the right set of KPIs. It was a great preview of what was ahead. 🔹 Full Loop: Four back-to-back interviews, each with its own focus: 🔸 1. Technical Skills This round tested my ability to work through open-ended product data problems with SQL and logical reasoning. From articulating assumptions to iterating on edge cases, it was about more than just code — it was about how you think under uncertainty. 🔸 2. Analytical Execution Here, I was asked to frame hypotheses, compare metrics trade-offs, and run through how I’d structure an analysis to support a product decision. I appreciated how grounded the questions were in real product scenarios — it truly felt like solving problems that matter. 🔸 3. Analytical Reasoning This was the most conceptual round — designing experiments, choosing the right analytical approach, identifying biases, and translating data into product insights. It challenged me to zoom out and explain the why behind every step. 🔸 4. Behavioral A chance to share examples of navigating ambiguity, collaborating across teams, learning from failure, and staying resilient. This round reminded me that the how is just as important as the what when it comes to impact. 📉 Rejections are tough — there's no sugarcoating that. But this process gave me clarity, confidence, and the kind of growth that doesn’t always come with a “yes.” I’m extremely proud of how far I went, and even more excited for what’s next. Onwards and Upwards 🚀 #Meta #DataScience #DataAnalytics #InterviewExperience #CareerJourney #TechInterview #JobSearch

876

15 Comments

LLMs are learning to think better by doing the opposite of humans. While we typically think fast first and only slow down when stuck, new research shows AI reasoning models perform best with the reverse approach. Researchers from UIUC and UC Berkeley just introduced AlphaOne (α1), a framework that teaches Large Reasoning Models like OpenAI's o1 to modulate their thinking speed at test time. The key insight? These models suffer from either overthinking (wasting compute on simple problems) or underthinking (rushing through complex ones). The solution is elegantly simple: α1 uses an "α moment" to scale the thinking phase and schedules slow-to-fast reasoning transitions using stochastic processes. During the initial phase, the model thinks slowly and carefully, then switches to fast thinking for efficient answer generation. This approach improved accuracy by up to 6% while reducing computational costs by 14%. What's fascinating is that this slow-first, fast-later pattern directly contradicts human cognition as described by dual process theory - suggesting that optimal AI reasoning may follow fundamentally different principles than biological intelligence. The implications go beyond efficiency gains: as reasoning models become central to AI applications, understanding how to optimize their thinking patterns could unlock new capabilities we haven't yet imagined. ↓ 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

144

13 Comments

What if there exists a 1:1 mapping between graphs <-> sequences of symbols? Will LLMs treat graphs just like sentences? My paper “Directed Graph Grammars for Sequence-based Learning” is dropping at [ICML] Int'l Conference on Machine Learning this year! TLDR: We establish a bijective mapping between graphs <-> sequences. We define the ideal properties for such a mapping, propose a grammar-based compression algorithm to learn the mapping, and prove those properties are satisfied. Then, we apply Transformers to generate the sequential descriptions directly. (We focus on DAGs due to a few technical considerations, but the theory in principle extends to general graphs too.) Directed acyclic graphs (DAGs) are used to represent everything from electronic circuits to neural network architectures, but they are hard to generate because there’s no single 𝘤𝘰𝘳𝘳𝘦𝘤𝘵 order in which to build their nodes and edges, leading to permutation-sensitive sequential descriptions that lead to brittle decoders. We introduce Directed Graph Grammar Embedded Derivations (DIGGED), which rewrites a DAG as a 𝒖𝒏𝒊𝒒𝒖𝒆 sequence of 𝘳𝘶𝘭𝘦-𝘣𝘢𝘴𝘦𝘥 𝘤𝘰𝘯𝘴𝘵𝘳𝘶𝘤𝘵𝘪𝘰𝘯 𝘴𝘵𝘦𝘱𝘴. We learn these rules by repeatedly mining common patterns, solving compatibility puzzles to derive maximally shared rewrite rules via a description length minimization objective, and iteratively compressing the provided data without losing information. Crucially, the DIGGED representation establishes a one-to-one problem mapping between DAG generation and sequence modeling, and we take full advantage by training Transformer-based decoders for generation and downstream optimization. On benchmarks spanning neural architectures, Bayesian networks, and analog circuits, DIGGED achieves perfect validity and uniqueness, substantial gains in predictive accuracy and Bayesian optimization performance over prior DAG decoders, and delivers an interpretable, compact representation that bridges graph-structured data with powerful sequence-based methods. By turning any DAG collection into a compact, principled 𝘥𝘦𝘴𝘪𝘨𝘯 𝘭𝘢𝘯𝘨𝘶𝘢𝘨𝘦, DIGGED opens the door to more reliable, scalable graph design and optimization across scientific and engineering domains. Co-authors: Orion Foo Gang Liu Wojciech Matusik Jie Chen Preprint: https://lnkd.in/e93hagP8 Code: https://lnkd.in/eHAhGCFA If this work interests you, let’s have a chat at ICML.

126

7 Comments

Been getting lots of questions about how to land an Applied Scientist / Research Scientist role — here’s what worked for me: 1️⃣ Master core coding patterns – Focus on Strings, Trees, Graphs, DP. Depth > quantity. 2️⃣ Know ML in breadth & depth – Classic ML + job-relevant deep learning topics. Understand the full ML pipeline — from large-scale training to real-world deployment. 3️⃣ Prepare behavioral answers like a scientist – Structure a few solid stories that map to company values. 🎯 Pro tip: Don’t wait for clarity — create it. Opportunities rarely arrive in perfect form. Apply boldly, explore broadly, and let momentum reveal your path. #appliedscientist #researchscientist #machinelearning #jobsearch #generativeai #careeradvice #datascience

195

3 Comments

The course on AI Agents (taught at UCSD CSE) is comprehensive, covering everything from classical planning and reinforcement learning to cutting-edge topics like multi-agent systems and human-AI collaboration. I could only find slides and suggested reading (no video, but perhaps someone uploaded it) Course Link: https://lnkd.in/gSknkQKG

71

Something smaller today: at the Institute for Experiential AI at Northeastern University my coworker Saumya Gupta and I have been working hard on bringing Kernel PCA methods into a generative AI model. During this, I discovered a noise-robust Kernel PCA method published a few years back, with apparently no python implementation. So, here it is! An implementation based of 'Exactly Robust Kernel Principal Component Analysis' by Jicong Fan and Tommy W.S Chow from IEEE 2019. 🧐 𝐖𝐡𝐲 𝐢𝐭 𝐦𝐚𝐭𝐭𝐞𝐫𝐬? Kernel PCA is a blazing fast and successful nonlinear data projection technique. The key weakness of kernel PCA is that it is extremely susceptible to outliers and out of distribution data. Robust Kernel PCA aims to resolve this by assuming there is noise in the data it's fit to and filtering out the real signal from data. 🔍 𝐖𝐡𝐚𝐭'𝐬 𝐢𝐧 𝐢𝐭? The repo contains a RKPCA class implemented in python using NumPy, PyTorch, and a comparison against scikit-learn's kernel PCA. It shows the explained variance for various data compression scales, and how to call the RKPCA class to fit, transform, and inverse transform data. 🛠️ 𝐑𝐞𝐩𝐨: 👉 https://lnkd.in/d4esYXfm If you're working on machine learning, generative AI, or data compression, give it a spin and let me know what you think! #MachineLearning #AIResearch #BuildInPublic #OpenSource

31

🚀 New Paper: Localizing Knowledge in Diffusion Transformers Understanding how knowledge is distributed across the layers of generative models is crucial for improving interpretability, controllability, and adaptation. While prior work has explored knowledge localization in UNet-based architectures, Diffusion Transformer (DiT)-based models remain underexplored in this context. In this paper, we propose a model- and knowledge-agnostic method to localize where specific types of knowledge are encoded within the DiT blocks. We evaluate our method on state-of-the-art DiT-based models, including FLUX, PixArt-alpha, and SANA, across six diverse knowledge categories. We show that the identified blocks are both interpretable and causally linked to the expression of knowledge in generated outputs. Building on these insights, we apply our localization framework to two key applications: model personalization and knowledge unlearning. In both settings, our localized fine-tuning approach enables efficient and targeted updates, reducing computational cost, improving task-specific performance, and better preserving general model behavior with minimal interference to unrelated or surrounding content. Overall, our findings offer new insights into the internal structure of DiTs and introduce a practical pathway for more interpretable, efficient, and controllable model editing. 🌐 Project page: https://lnkd.in/dFWi6gJ2 📄 Paper: https://lnkd.in/dhBEF5Da Joint work with: Samyadeep Basu, Keivan Rezaei, Zihao Lin, Sayan Nag, Soheil Feizi

159

23 Comments

Excited to share some new research on efficient learning for small LLMs! Aditri Bhagirath, Moritz Bolling and I found that gradient updates to language models are inherently low-rank even in small model space, observing that rank 1 decompositions (e.g. using DoRa or LoRa) yield negligible performance deficits. Distilled, high-speed models are becoming more and more popular for their speed and cost benefits, and this finding suggests that fine-tuning of such models can be quick, accessible, and cheap using low-rank gradient updates during training. https://lnkd.in/gPDaKnfR

65

🔍 Accelerating the Future of AI: AI Agent Dictionary Explained Today, I’m excited to share a concise yet powerful AI Agent Dictionary that highlights key concepts shaping the next generation of intelligent systems. Whether you're working on autonomous systems, multi-agent environments, or next-level conversational AI, understanding these foundational terms is essential for building smarter and more reliable AI solutions. This dictionary covers a wide spectrum of modern AI concepts — from Autonomous Agents and Goal-Oriented Architectures to Neural-Symbolic AI, Reinforcement Learning, Knowledge Graphs, and Explainable AI. Each term provides clarity on how intelligent agents perceive, reason, learn, and interact within dynamic environments. ✨ Why this matters As AI continues to evolve, agents are becoming more proactive, collaborative, and context-aware. These concepts empower developers, researchers, and organizations to design systems that: Work autonomously with minimal human oversight Learn continuously from real-world interactions Coordinate efficiently in multi-agent settings Make transparent and explainable decisions Optimize actions based on goals and utility 🌐 From Federated Learning that protects data privacy to Swarm Intelligence that mirrors nature’s collective problem-solving, each concept represents a building block for future-ready AI applications across industries such as healthcare, robotics, cybersecurity, education, and enterprise automation. 📘 Whether you're a student exploring AI, a developer building intelligent applications, or an AI professional advancing your expertise, this dictionary provides a valuable reference for navigating the rapidly evolving agent-based ecosystem. If you found this useful, feel free to save, share, or reach out to discuss anything AI-related! #AI #ArtificialIntelligence #AIAgents #MachineLearning #DeepLearning #MultiAgentSystems #LLMs #ReinforcementLearning #ExplainableAI #FutureOfAI #Innovation #Tech

19

🎉 Big news: PrepVector is officially teaming up with StrataScratch! I’m excited to announce that we’ve officially partnered with @StrataScratch to help data professionals go beyond interview prep and build real-world impact in their careers. At PrepVector, we run live cohort-based courses that focus on upskilling data professionals. We teach them to think like product leaders, run rigorous experiments, and apply ML to solve business problems — the kind of work that drives real outcomes. StrataScratch has long been a go-to resource for upskilling programming and analytical skills. Several of our learners already use it and I have personally been using the platform for 5+ years now (since their first MVP launch!) Now we are teaming up to help learners grow faster, build industry skills, and own their impact at work. Thank You StrataScratch! I can’t wait to build with you!   This is just the beginning — and I couldn’t be more excited for what’s ahead. Here’s to building something special for the data community. 💙 Check out the offerings here: https://lnkd.in/eRnihgJW https://www.prepvector.com

78

16 Comments