Joan Xiao

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

968

21 Comments

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

DGX Spark seems like a cool product that simplifies AI dev such that debugging things can be easier. I.e. being able to look at the generated videos��from ablation experiments without needing an scp. One important question is what is the time to compile PyTorch from scratch though? DGX Spark only has 20 tiny ARM cores (with 10 out of the 20 being even smaller ARM cores). It already takes a decent chunk of time compiling on dual socket xeon/eypc and dual socket DC grade grace CPUs.

27

Large Language Models can't access your private data unless you help them do so. In this tutorial, Maham shows you how to use Langbase's Memory Agents for this. You'll learn how to securely let LLMs chat with your docs by creating and integrating your own memory agent. https://lnkd.in/gZkYWyCw

387

4 Comments

Building agentic applications is complex, but it doesn’t have to be slow. In this blog, Samraj Moorjani shows how to go from idea to measurable results in hours using the Claude Agent SDK with MLflow for observability and evaluation. What you’ll learn: 🔸 Rapid prototyping with the Claude Agent SDK to validate ideas quickly.​ 🔸 Automatic tracing with MLflow’s @mlflow.anthropic.autolog() for zero-instrumentation visibility into every agent action.​ 🔸 Objective iteration using mlflow.genai.evaluate() with custom scorers and judges to track quality improvements.​ 🔗 Dive in: https://lnkd.in/eAGpbs5b ​ #MLflow #agentic #claudeagent #oss #opensource

161

2 Comments

Demo: how to surface and use Unity Catalog Metric Views in Databricks AI/BI — to enable self-serve access to trusted business metrics. See how to set up central governance with row-level security and column masking, and more.

403

8 Comments

We’re excited to welcome Ray to the PyTorch Foundation 👋 Ray is an open source distributed computing framework for AI workloads, including data processing, model training and inference at scale. By contributing Ray to the PyTorch Foundation, Anyscale reinforces its commitment to open governance and long-term sustainability for Ray and open source AI.    “The PyTorch Foundation is committed to fostering an open, interoperable, and production-ready AI ecosystem. By bringing Ray under the PyTorch Foundation umbrella, alongside projects like vLLM and DeepSpeed, we are uniting the critical components needed to build next-generation AI systems. Ray’s inclusion strengthens our collective mission to support developers with the tools to efficiently train, serve, and deploy AI models at scale.” - Matt White GM of AI at The Linux Foundation Foundation and Executive Director of the PyTorch Foundation “With Ray, our goal is to make distributed computing as straightforward as writing Python code. Joining the PyTorch Foundation helps us stay true to that mission, ensuring Ray continues to be an open, community-driven backbone for developers and their organizations.” - Robert Nishihara, co-founder of Anyscale ➡️ Read the announcement: https://lnkd.in/e3e4mjds #Ray #PyTorchCon #PyTorch #AI #OpenSource #AICompute #DistributedComputing #AIWorkloads #DataProcessing #ModelTraining #Inference

667

5 Comments

Accelerate topic modeling with GPUs in Snowflake ML. Large-scale topic modeling does not have to take hours. With native GPU acceleration in Snowflake ML, you can speed up pandas, scikit-learn, UMAP, and HDBSCAN workflows using just two lines of code. In this quickstart, you will learn how to: ✅ Run GPU-enabled Snowflake Notebooks ✅ Enable NVIDIA cuDF & cuML with zero refactoring ✅ Process 500K documents in under a minute ✅ Turn multi-hour ML pipelines into rapid iterations If you’re working with massive text datasets, this is a must-try workflow for faster experimentation and production-ready ML. Get started: https://lnkd.in/gN3Gw9UP

86

2 Comments

Quartr API datasets are now available via Snowflake. Access structured earnings data – live audio, real-time and historical transcripts, filings, slide decks, and event summaries. Pull Quartr data directly into your existing Snowflake workflows and augment your datasets. No new pipelines, no separate API integration. Build with data that drives decisions.

75

If your team is building with LLMs, you know the value of clean, retrievable data. That's why we're teaming up with Unstructured to streamline one of the hardest parts of AI development. This integration removes friction across the AI stack—from unstructured ingestion to semantic retrieval—so your teams can focus on building. Read more from our Product Marketing Managers, Rini Vasan and Jim Allen Wallace, and Unstructured’s Head of Developer Relations, Maria Khalusova here: https://lnkd.in/e98QqXAB

43

Meta’s Llama 4 Large Language Models are now available in Snowflake Cortex AI! Llama 4 introduces Meta’s first Mixture-of-Expert (MoE) architecture for faster, more efficient inference, helping customers build high-performing enterprise gen AI apps and deliver personalized experiences inside Snowflake. Now available: - Llama 4 Maverick: A powerful, natively multimodal model, built for efficient single-host inference. - Llama 4 Scout: A fast multimodal model with a 10 million token context window, optimized to run on a single GPU. With Llama 4 models in Cortex AI, customers can access leading-edge AI to build intelligent apps and data agents directly in Snowflake. Learn more: https://lnkd.in/g4t7UGB8

824

19 Comments

Data pipelines for AI often start with structured data. Ray Data now supports traditional tabular workloads alongside multimodal and accelerator-native use cases. At Ray Summit 2025, Alexey Kudinkin from Anyscale will present Structured Data Support for Ray Data. What you will learn: - Key new features for tabular data: joins, expressions, and data preprocessors - The architectural changes that made these features possible - Performance improvements across Ray versions and a look at the future roadmap Nov 3–5 at the San Francisco Marriott Marquis Register now: https://lnkd.in/g4PdxSim

40

1 Comment

Through the new Elastic Inference Service (EIS), now available on Elastic Cloud, we’re bringing GPU-accelerated inference natively to Elasticsearch. EIS combines Elasticsearch’s vector and semantic search capabilities with GPU-powered inference for text embeddings, semantic reranking, and large language models. Developers can now build semantic search, GenAI, and end-to-end RAG applications faster and more efficiently. Learn more: https://go.es.io/4o73Tl3

161

We’ve worked with Heroku to make Pydantic AI integrate seamlessly with Heroku Managed Inference & Agents. This combination lets you: * Build agents in a familiar Pythonic style, with Pydantic’s data validation at the core * Deploy and run them on Heroku’s fully managed infrastructure * Use standard protocols like Model Context Protocol (MCP) and Agent-to-Agent (A2A) for interoperability The blog post walks through creating and deploying a Pydantic AI agent on Heroku in just a few steps. Read the guide here: https://lnkd.in/epwR98kH

67

2 Comments

AI works best with a human in the loop 🔁. In this episode, Ion Stoica, Co-founder of Anyscale and Databricks, explains why the most successful AI applications like code assistants and customer support thrive when humans are still reviewing and approving the output. He breaks down how this model applies to data teams using tools like Coalesce: boosting productivity without sacrificing accuracy or control. 🎥 Watch the full recording to hear Ion’s take on AI, reliability, and the future of data workflows: https://lnkd.in/gWzQRggW

19

🚀 Qwen3-VL Tech report is now out on arXiv! From pretraining to post-training, architecture to infra, data to evaluation — we’ve packed in the details for anyone building on vision-language models. 🔥 3 models >1M downloads in just over a month 🏆 Qwen3-VL-8B leads with 2M+ downloads 📚 Built on the shoulders of Qwen2.5-VL (2800+ citations in <10 months!) Check out the paper for insights, baselines, and future directions. Let’s keep pushing VLMs forward — together. https://lnkd.in/gV-kPFTf

818

13 Comments

Quantum needs data—not just qubits. aqora’s new public datasets hub marks a practical step forward for the field. By creating a central space to upload, access, and collaborate on quantum-specific datasets, they’re addressing a quiet but critical challenge in the ecosystem: reproducibility and shared benchmarks. Why does this matter? Quantum machine learning and benchmarking rely on consistent, high-quality data—but until now, much of it has been scattered, inconsistent, or locked in silos. This hub offers: ➡️ Versioned, well-structured datasets ➡️ Integration with tools like pandas and polars ➡️ Options for both public and private data sharing ➡️ A base for teaching, testing, and real-world use cases As the field matures, tools like this can help shift the conversation from isolated experiments to collaborative progress. 🔗 Full article in the comments. #QuantumComputing #QML #OpenData #QuantumResearch #TQI #TheQuantumInsider #Aqora #EcosystemBuilding

79

5 Comments

Tuning Spark, Ray, or Dask for ML isn’t one-size-fits-all. In a new Domino Blueprint, Principal Solution Architect Sameer Wadkar shows how API-driven scaling complements autoscaling to right-size clusters on demand—cutting idle spend and speeding bursts for GPU/I/O-heavy work. Practical steps, a lightweight Python client, and Helm install included. If you care about FinOps and MLOps, this is for you. Read the blueprint: https://lnkd.in/ga_qJkhC

7