Wenqian Yan

🚀 Excited to share our Final Semester Project: "Simulation of Augment in LLM Models" a Multi-agent AI assistant built from scratch and deployed end-to-end on the cloud. It combines LLM agents, real-time APIs, RAG-based memory, and a full AWS MLOps stack for deployment, orchestration, and monitoring ✨ Standout capability: Adapts to personal tone formal, casual, or conversational by learning from chats, emails, and behavior. Key Functionalities include, 🧠 Maintains long-term memory using Retrieval-Augmented Generation (RAG) ✍️ Adapts to your tone and language using Neural Style Transfer 📧 Automatically reads inbox, summarizes emails, drafts replies, and even reads and summarizes links 🌐 Pulls real-time information from the web to answer live queries 🗓️ Scans for meeting invites in emails, schedules them in Google Calendar + Zoom, reflects updates, and answers from transcripts 📋 Creates, tracks, and deletes tasks from emails or chat threads 💬 Integrates with Google Chat, responding in the same tone as the user 🥗 Suggests meals and tracks sentiment for mental wellness 💡 LLMs Used and Fine-Tuned: Integrated and fine-tuned a diverse set of powerful LLMs including GPT-4o, Gemini-2.0-flash-001, Gemini 1.5 Pro, Claude 3.5, LLaMA 3.3b, DeepSeek R1. 🧠 Trained models using Hugging Face and Vertex AI for tasks like memory augmentation, summarization, calendar reasoning, finetuning. Built specialized agents (email, calendar, chat, web, tasks), coordinated by a central controller. A tone adaptation agent personalized responses based on user style. Used various APIs, LangChain for orchestration, FAISS for memory, and PGVector + Redshift for embedding storage enabling rich, contextual interactions via a real-time Streamlit UI. 📊 Evaluation Tools:  Tested models using Ifeval, HaluEval, GEval, TruthfulQA, Accuracy, and MMLU to assess correctness, hallucination resistance, consistency, and truthfulness. Gemini and GPT-4o consistently led with 88–95% accuracy on tasks involving memory, reasoning, and summarization. ⏱️ Latency Performance: Gemini led with 0.45s response time, closely followed by GPT-4o at 0.48s. DeepSeek R1 and LLaMA 3.3b lagged, with DeepSeek taking 36.49s for a simple time query. Claude 3.5 showed moderate, steady performance. Tests spanned real-world tasks like email summarization, task automation, tone adaptation, web-based responses etc. Built a cloud-native MLOps pipeline on Amazon Web Services (AWS) for model training, inference, and monitoring. Used S3, Glue, Lambda, SageMaker, Redshift, RDS, and PGVector for orchestration and storage, with EventBridge, Step Functions, and CloudWatch for automation. Delivered real-time insights via QuickSight and Streamlit. 🌐 Why It Matters: Blending LLM agents, memory, and MLOps, our assistant delivers clarity, automation, and a human touch in an overwhelming digital world. #AI #LLM #GPT4 #Gemini #Claude #LangChain #MLOps #AWS #Streamlit #FineTuning #AIassistant #RAG #Teamwork #Leadership

45

2 Comments

As a PhD student in Machine Learning Systems (MLSys), my research focuses on making LLM/GenAI serving and training more efficient. Over the past few months, I’ve come across some cool papers that keep shifting how I see this field. So, I put together a curated list to share with you all: https://lnkd.in/gYjBqVPt This list has a mix of academic papers, tutorials, and projects on GenAI systems. Whether you’re a researcher, a developer, or just curious about GenAI Systems, I hope it’s a useful starting point. The field moves fast, and having a go-to resource like this can cut through the noise. So, what’s trending in GenAI systems? One massive trend is efficiency. As models balloon in size, training and serving them eats up insane amounts of resources. There’s a push toward smarter ways to schedule computations, overlap communication, compress models, manage memory, optimize kernels, etc. —stuff that makes GenAI practical beyond just the big labs. Another exciting wave is the rise of systems built to support a variety of GenAI applications/tasks. This includes cool stuff like: - Reinforcement Learning from Human Feedback (RLHF): Fine-tuning models to align better with what humans want. - Multi-modal systems: Handling text, images, audio, and more. - Chat services and AI agent systems: From real-time conversations to automating complex tasks, these are stretching what LLMs can do. - Edge LLMs: Bringing these models to devices with limited and heterogeneous resources, like your phone or IoT gadgets, which could change how we use AI day-to-day. The list isn’t exhaustive, so if you’ve got papers or resources you think belong here, drop them in the comments.

843

24 Comments

Want to train LLM to reason right on your mac? Check out my article on Reinforcement Fine-Tuning LLMs with GRPO on Apple silicon using MLX framework. I remembered vividly when DeepSeek-R1 burst onto the scene this year, it dominated discussions over Chinese reunion dinners, AI stocks crashed, and friends I knew (including me) were busy asking it to reason their Bazi. A key idea introduced along with Deepseek R1 was Group Relative Policy Optimisation (GRPO), which helps to incentivise reasoning capability in LLMs via reinforcement learning. Since then, there have been courses such as Predibase's course on implementing GRPO, and improvements to the algorithm such as Mistral AI's Magistral LLM which was trained on a more efficient GRPO algorithm. Following the course by Predibase hosted on DeepLearning.AI, I have written an article where I explain step by step each component of GRPO equation, with a worked example to illustrate the algorithm. If you have on hand an Apple silicon computer, I have also included a Jupyter notebook which implements GRPO using Apple's MLX framework. Credits to Awni Hannun for creating the base notebook for LoRA Fine-Tuning with MLX LM. Full disclosure that I created the article and the notebook with the assistance of Google Gemini. If you noticed any errors with either the article or the notebook, please let me know or file a pull request.

889

39 Comments

I was in San Francisco✈️ this weekend and got the opportunity to review the codebase of a very early-stage startup🔥. It was interesting to see how products built on LLM wrappers actually run in real environments. As someone who works in LLM systems, I always assumed things like KV-cache, prefilling, and TTFT were widely understood. But many teams, especially early-stage ones, don’t always think deeply about these details, even though they make a big difference in performance. ⚙️🚀 One example I saw was passing the full conversation history to the model on every request. It works, but it forces the model to recompute everything from scratch, adding unnecessary latency. If you're using inference APIs, reusing the 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 from previous requests can give you a huge boost. With OpenAI, this is done through 𝐓𝐡𝐫𝐞𝐚𝐝𝐬, which creates a session on their side where your KV cache is stored. This reduces prefilling time and also cuts down network overhead because you're sending less data each time. ⏱️📉 I believe many of you are already using this. If not, I recommend reading about 𝐓𝐡𝐫𝐞𝐚𝐝𝐬 and 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 reuse, it’s a small change that can make your system much faster. 📚💡 And if you rely heavily on inference and latency is becoming a pain point, I’d be happy to look at your implementation and share suggestions to the best of my knowledge. 🤝🚀 #LLMSystems #InferenceOptimization #KVCache #OpenAIThreads

238

3 Comments

In this week’s AI infra chat, we sit down with Seiji Eicher, Distributed LLM Inference Engineer at Anyscale to unpack how large MoE models are actually served in practice, what it looks like to work deep in the LLM inference stack & more 🕐 Timestamps: 𝟬𝟬:𝟬𝟬 Why MoE comes up in modern AI infra 𝟬𝟬:𝟮𝟯 What Mixture of Experts actually changes inside a model 𝟬𝟭:𝟮𝟰 Why expert parallelism exists (and what becomes inefficient without it) 𝟬𝟮:𝟱𝟳 When MoE helps in practice 𝟬𝟰:𝟮𝟯 How Ray Serve fits into real inference stacks 𝟬𝟳:𝟬𝟳 What a Distributed LLM Inference Engineer works on day to day 𝟭𝟭:𝟭𝟵 Where the space is headed and how to learn more If you want to go deeper, Seiji gave a Ray Summit talk that dives into production-grade serving of MoE models, including KV-cache optimization, parallelism strategies, and multi-node orchestration with Ray Serve and vLLM. Check out the talk linked in the comments 👇

60

3 Comments

Making massive AI models accessible and performant is one of the biggest challenges for AI services. That's why I'm thrilled to announce our latest collaboration with the open-source community: a completely re-architected vLLM backend for Google Cloud TPUs. We’ve created a unified path that allows developers to run models written in both PyTorch and JAX through a single, highly-optimized inference engine on TPUs. This means you can now leverage vLLM on TPUs with zero code changes and get the full performance benefits of TPUs. With the new backend, we're seeing up to 5x higher throughput on popular models when compared to our initial prototype just a few months ago. This is a game-changer for lowering cost and improving user experiences for demanding AI applications. Even more, it creates new capabilities as developers and makers can now leverage higher quality model and agentic workflows with tremendous efficiency. This unified vLLM stack lays the groundwork for even more innovation with the coming GA of Ironwood, our next-generation TPUs. And it would not have been possible without our deep partnership with the incredible vLLM open-source community. Dive into the technical details in the official blog post, and try it out today on Google Cloud. https://lnkd.in/gGH9F4fA #GoogleCloud #TPU #AI #LLM #Inference #vLLM #PyTorch #JAX #OpenSource

787

6 Comments

Finally take some time to do a detailed dive into LLVM's instruction selection legalization phases. The purpose of legalization is fairly intuitive — make IR ops/types target-friendly — but it's interesting to see the mechanics and how real backends like AMDGPU interact with the process. These notes are mostly focused on sorting out the flow and structure of legalization. 👉 https://lnkd.in/gcS5P4M7

40

Super excited to share our latest work (even as I wrap up graduation 😁)! We've just released the preprint of our new paper on semantic recall in long-context code reasoning. In this work, we highlight how LLMs' "lost in the middle" effect—where performance drops when key information appears mid-context—worsens on a novel task that pushes LLMs to recall specific semantic aspects of code (e.g., outputs of individual statements). This paper is packed with interesting findings, backed by exhaustive experiments across 10 frontier LLMs (including GPT-4.1 and open-source models up to 256 billion parameters), and connects these empirical results to hypotheses about how current LLMs reason over long contexts. Huge thanks to Adam Štorek for leading the project, to my incredible collaborators Samira Hajizadeh & Prashast Srivastava, and Prof. Suman Jana for his guidance throughout.

148

12 Comments

Google Research Director Denny Zhou, who founded the LLM Reasoning Team at DeepMind, recently gave a great talk at Stanford’s CS25 class: Large Language Model Reasoning. This was one of the most intuitive talks about reasoning model training and application. He outlined new directions for training models to handle questions where answers aren’t easily verifiable. My team is currently working on an enterprise LLM project focused on using reasoning to extract relevant context and generate responses for unverifiable cases. It’s a tough but promising area, and it’s an interesting time to be building. [Link to Denny Zhou's talk: https://lnkd.in/gdJ6Mzi8]

1,026

8 Comments

Our team published a deep dive on how we built our in-house AI data agent to work at real data-platform scale. The agent reasons over 600+ PB across ~70k datasets, using natural language to help engineers, product teams, and researchers get trustworthy answers grounded in correct metadata, ownership, and access controls. The real challenge wasn’t just LLMs, it was building tight integrations with our data platform: table-level knowledge, catalogs, governance, and execution layers, so answers are accurate, fresh, and auditable. Under the hood: Codex-powered reasoning on top of structured metadata and platform context. Learn more on our Eng Blog: https://lnkd.in/gvCCbhrk If you're interested in building bleeding-edge AI tools for data analysis, please apply today: https://lnkd.in/g7DUrUNz

270

17 Comments