Eyal Lantzman

A lot of agentic pipelines are using frontier LLMs for tasks that don't need them. Entity recognition, intent classification, policy checking - these are extraction tasks, not reasoning tasks, and the cost and latency show it. Ash Lewis, Co-founder and CEO of Fastino Labs, is presenting at Coding Agents on March 3rd on how GLiNER2 - a 205M parameter encoder model - handles four core extraction tasks in a single forward pass, with deterministic outputs and no prompt engineering. The case he's making is straightforward: reserve the LLM for reasoning and generation, and stop paying frontier prices for structured extraction. Computer History Museum, Mountain View. March 3rd. Tickets are nearly gone - grab yours before it closes: https://lnkd.in/giDNa7fG

26

Your LLM agent is giving outdated answers, but your monitoring is silent. Yuval Gorchover's article on the Upstream Mentality dives into how "AI failures" are often camouflaged and how to prevent them with proactive data lineage mapping and risk assessment.

1

Let's talk parsing tables. Two days ago we launched ParseBench, the first benchmark for measuring document OCR accuracy for AI agents, with 5 new metrics. In this deep dive, we break down TableRecordMatch (GTRM), our metric for evaluating complex tables with merged cells, hierarchical headers, and multi-page spans. Instead of comparing tables cell by cell, TableRecordMatch evaluates them the way downstream systems actually consume the data, as records keyed by column headers. Column reordering doesn't tank your score. Transposed headers and dropped columns do. Read our full blog on ParseBench, access our source code, and get access to the benchmark dataset on Hugging Face all here: https://lnkd.in/epksZ3Nk

56

3 Comments

🔥 ENGINEERING UPDATE: Redpanda AI SDK for Go is now #opensource As part of our #AgenticDataPlane (ADP), we're building managed and self-hosted AI agents. These agents need to work in production, not just in demos. They also need to be observable, resilient, flexible across providers, and easy to test without crossing your fingers before every deploy. 🤞 We checked existing #Go options and found useful pieces, but not the full package. So we built one. Today, we’re open-sourcing it. Read all about it and how to get started👇 https://lnkd.in/dDv8aXbT

30

POML: Prompt Orchestration Markup Language Introduction As Large Language Models (LLMs) become increasingly central to modern applications, developers face growing challenges in prompt engineering. While LLMs can produce powerful results, the lack of structure and consistency in prompt design often leads to fragile, hard-to-maintain workflows. POML (Prompt Orchestration Markup Language) was created to solve these issues. It introduces a structured, modular, and versatile approach to building prompts—empowering developers to design reliable, scalable, and sophisticated LLM applications. Key Features of POML 1. Structured Prompt Design Instead of writing long, unorganized text prompts, POML allows developers to break down prompts into modular components. This ensures clarity, reusability, and easier debugging. 2. Data Integration Made Easy POML supports seamless integration of dynamic data sources—from JSON and databases to external APIs. This eliminates the need for messy string concatenation and improves maintainability. 3. Presentation Flexibility Different tasks or users may require different prompt formats. POML provides a way to switch presentation variations without rewriting the entire prompt logic. 4. Tooling and Ecosystem Support With extensions available for VSCode, PyPI, and npm, POML integrates smoothly into existing developer workflows. It also offers testing and validation tools to ensure prompt quality. Why POML Matters • Consistency – Reduce unpredictable LLM behavior by enforcing structured prompt patterns. • Collaboration – Teams can work on modular prompt files with version control, making changes safer and more transparent. • Scalability – Ideal for enterprise-level applications where prompts must be reused, monitored, and updated frequently. • Experimentation – Quickly test variations of the same prompt structure without breaking the entire pipeline. Use Cases 1. Customer Support Standardize multi-turn conversations with structured templates. 2. Content Generation Create flexible article outlines, summaries, or scripts that adapt to dynamic input data. 3. Enterprise Applications Integrate with existing data pipelines (CRM, analytics, finance) while keeping prompts consistent. 4. Research & Experimentation Quickly test how variations in constraints, roles, or examples affect model output. Conclusion POML represents a step forward in prompt engineering—offering the structure and flexibility developers need to scale LLM-powered applications. By combining modular design, data integration, and ecosystem support, POML makes it easier to build prompts that are reliable, reusable, and production-ready.

1

Examining how managers should deploy AI in sequential team workflows finds that optimal AI deployment is stochastic, replaces early and late positions in the workflow, and results in lower wage inequality, from Xienan Cheng, Mustafa Dogan, and Pinar Yildirim https://lnkd.in/ewSrbPmH

53

vLLM has reached 66k+ GitHub stars and millions of downloads in just over two years, and is widely adopted as an open source inference engine. 🎥 At Ray Summit, we sat down with Simon Mo, co-lead of the vLLM project, to discuss how vLLM is built, architectural choices , how Ray and vLLM work together, what’s next for the project and much more! ⏱️ Chapters & Timestamps 𝟬:𝟬𝟬 Overview of vLLM 𝟬𝟭:𝟬𝟭 PagedAttention & Early Architectural Decisions 𝟬𝟮:𝟭𝟭 Why vLLM Adoption Is Accelerating 𝟬𝟰:𝟮𝟴 How vLLM and Ray Work Together 𝟬𝟳:𝟬𝟬 The State of vLLM Today 𝟭𝟬:𝟬𝟵 Simon Mo’s Open Source Journey 𝟭𝟮:𝟮𝟴 Advice for AI Builders & Contributors

132

1 Comment

Common GPU Bottlenecks During Fine-Tuning and How to Avoid Them in 2025 Fine-tuning LLMs and diffusion models is expanding fast, but many teams still lose 30-50% of training time because of avoidable GPU bottlenecks. These slowdowns increase cloud spend, delay iterations & reduce model throughput. Knowing these bottlenecks early helps teams train faster and scale more efficiently. 1. Underutilized GPUs GPUs often operate at only 40-60% utilization because the data pipeline is slow to keep up. This forces GPUs to wait between batches, creating wasted compute hours. Many teams misdiagnose this as poor GPU performance when the real issue is upstream in data loading.  Fix: Use parallel data loaders, prefetching, NVMe storage and larger batch sizes to keep the GPU saturated. 2. VRAM Overflow Large models like LLaMA 3, Qwen & Mistral often exceed VRAM limits, which pushes computation onto the CPU. If this happens, training can slow down by as much as 40× because CPUs cannot handle GPU-level tensor operations. This not only hurts performance but also increases training instability.  Fix: Use gradient checkpointing, QLoRA (4-bit or 8-bit) and higher-VRAM GPUs like A100 80GB, H100 or L40S. 3. CPU Bottlenecks Even the fastest GPUs perform poorly when paired with underpowered CPUs that cannot keep up with preprocessing, tokenization or batch preparation. These CPU delays create idle time, making GPU performance appear inconsistent. As models grow, poor CPU throughput becomes an even bigger constraint.  Fix: Choose GPU servers with balanced CPU ratios, enable multi-threading and move heavy preprocessing to async workers. 4. Disk I/O Limitations When datasets sit on slow disks, the GPU repeatedly pauses to wait for data reads. This issue becomes even more noticeable with large datasets, high-resolution image pipelines where read operations dominate. The result is lower GPU utilisation & unnecessary delays.  Fix: Use NVMe SSDs, keep datasets local and avoid slow network-attached storage during training. 5. Wrong Precision Modes Many teams train in FP32 even though it offers no extra accuracy for most fine-tuning tasks. This wastes compute, restricts batch sizes and increases VRAM pressure, slowing down training overall. Lower precision modes deliver equal performance with dramatically faster throughput.  Fix: Use BF16/FP16 for most workloads and apply INT8/4-bit quantized fine-tuning when mem is tight. Want Faster Cost-Efficient Fine-Tuning?😄 If you want high-performance GPUs with predictable pricing, reservation options & architecture designed to remove these bottlenecks, Dataoorts is optimised exactly for fine-tuning. You get stable A100, H100 and L40S compute, fast provisioning and cost-saving reservation plans built for long training runs. Scale your fine-tuning with faster infrastructure, transparent pricing and smarter GPU planning. #AI #MachineLearning #LLM #GPUCloud #DeepLearning #FineTuning #A100 #H100 #Dataoorts #NVIDIA #GPURental #on-demand_cloud

4

𝗜𝗖𝗬𝗠𝗜 🚀 New LLM-as-judges Integration: TruLens Trace Evaluation in MLflow We are excited to announce the TruLens integration for MLflow! This expands our third-party scorer framework, which already supports DeepEval, RAGAS, and Phoenix—an ecosystem with 32M+ monthly PyPI downloads. In agentic workflows, a correct final answer can mask a flawed plan, redundant tool calls, or broken reasoning. To build reliable and trustworthy agentic systems, you must evaluate the execution traces. Developed by the TruLens team at Snowflake, this integration adds 10 scorers to MLflow that analyze the full span tree: 🔹 𝗚𝗼𝗮𝗹-𝗣𝗹𝗮𝗻 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁: Evaluates strategy and tool selection. 🔹 𝗣𝗹𝗮𝗻-𝗔𝗰𝘁𝗶𝗼𝗻 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁: Checks for plan adherence and valid tool calling. 🔹 𝗛𝗼𝗹𝗶𝘀𝘁𝗶𝗰 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁: Grades logical consistency and execution efficiency. You can now mix agent trace scorers, RAG scorers, and content quality scorers in a single 𝚖𝚕𝚏𝚕𝚘𝚠.𝚐𝚎𝚗𝚊𝚒.𝚎𝚟𝚊𝚕𝚞𝚊𝚝𝚎() call. 🔗 Full technical details: https://lnkd.in/eACHyxaK #MLflow #TruLens #LLM #GenAI #LLMOps #AIObserveability

65

Just open-sourced LangGraphGo 🦜🔗 The AI infrastructure powering Quizlink is now available on GitHub. LangGraphGo lets you build resilient AI agents as graphs with: → Real-time streaming → Graph visualization (Mermaid, DOT, ASCII) → State checkpointing → Langfuse observability → Works with OpenAI, Anthropic, Google AI If you're building AI in Golang, hope this helps. 🔗 https://lnkd.in/gkJbhUMN #opensource #golang #AI #langgraph #langgraph

15

By caching past keys and values, inference shifts from quadratic to linear complexity, cutting latency by ~70% and delivering 3–4× faster responses for long contexts. But speed comes at a price: massive VRAM usage, making memory the real bottleneck behind context window pricing. The takeaway? LLM inference today is a memory vs cost trade-off, and KV cache design is shaping everything from API economics to next-gen attention architectures. If you’re building or deploying LLMs at scale, this is one diagram worth internalizing. #LLM #GenerativeAI #AIInfrastructure #KVCache #MachineLearning #Inference #AIEngineering

2

Alibaba has introduced AgentScope, an open-source Python framework for building multi-agent applications. The framework provides visual tools for creating agents with support for MCP integration, memory, retrieval-augmented generation, and reasoning. It works with any LLM and includes real-time steering capabilities.

2

1 Comment

We just published a post walking through how to build a lightweight CLI AI agent in Go that runs on Hayride — an open-source secure AI runtime for LLMs, sandboxed code execution, and orchestrating agentic workflows. Our example uses: ⚙️ Go + TinyGo to compile the CLI to WebAssembly 🏗️ Hayride’s AI morphs (prebuilt Wasm components for agents, runners, tools, and models) 🏖️ WebAssembly Interface Types (WIT) and the Component Model for strict sandboxing and composability ⚒️ WAC for composing Wasm components into a deployable agent The post covers: ⚙️ Setting up the project and WIT dependencies ⌨️ Implementing a CLI in Go that streams prompts/responses to an LLM 🧩 Composition with Hayride’s prebuilt morphs 🚀 Deploying the resulting morph to Hayride If you’re curious about mixing Go, LLMs, and WebAssembly in a secure, composable AI environment — or just want to see an end-to-end example of building a CLI AI Agent — check it out: 📖 Read the post: https://lnkd.in/g-Wbs2Cb We would love feedback from other developers experimenting with WebAssembly + AI.

2

🚀 948x Faster? Google Just Solved the Biggest Headache in Generative Retrieval. If you’ve ever worked with LLMs for recommendation systems, you know the struggle: Generative Retrieval (GR) is brilliant for accuracy, but it's traditionally a total snail on hardware. The bottleneck? Tries (Prefix Trees). While we’ve used them for years to mask invalid tokens, they are a nightmare for GPUs and TPUs. They force irregular memory access and mess with static computation graphs (XLA). Basically, you have a Ferrari engine (the LLM) being fueled by a pipette. Enter STATIC. Google AI just dropped a new framework called Sparse MaTrix Framework for In-accelerator Constrained Decoding. Instead of clunky trees, it uses sparse matrix operations to handle constraints. The TL;DR on why this matters: ◾Insane Speed: We’re talking up to 948x faster decoding. 🏎️ ◾Hardware Native: It keeps the entire process on the accelerator (GPU/TPU), cutting out the CPU-to-Accelerator "lag." ◾Scale: You can now run Generative Retrieval across millions of items without your latency budget exploding. ◾The Bottom Line: This moves LLM-based search from a "cool research project" to a "high-performance reality" for massive industrial recommendation engines. Check out the full breakdown here: https://lnkd.in/eN3N_YrU #GoogleAI #MachineLearning #GenerativeAI #LLM #SearchTech #Innovation

1

As LLMs and agents move into production, a new class of failures is showing up: Not hallucinations - execution drift, unsafe tool calls, and silent workflow breaks. Most teams still debug these with logs and prompt tweaks. We’re experimenting with a different approach: Treat AI systems like distributed systems - trace decisions, classify failure modes, and firewall risky actions at runtime. We are curious: Where do your agents fail most often - tools, data, reasoning, or orchestration? #LLMOps #AIAgents

2

Is TOON Going to Be the Next Feed Format for LLMs, or Even Replace JSON? (It’s for flat JSON data, not nested ones.) As LLMs evolve, efficiency in data representation becomes crucial. That’s where TOON (Token-Oriented Object Notation) steps in, a compact and LLM-friendly format designed to make data handling faster and cleaner. Why TOON stands out: -Uses less data than formatted JSON (especially in large arrays). -Built-in guardrails with explicit lengths and fields. -Cleaner syntax, no extra braces or quotes. -Indentation-based structure like YAML. -Handles tabular arrays efficiently, keys declared once, data flows as rows. Note: TOON works best with flat JSON data. If your data is nested, converting it directly into TOON can make it heavier and less efficient. Always flatten your JSON before using TOON. Could TOON become the next data format standard for AI models? #TOON #TOONFormat #TechInnovation #JSONAlternatives #LLMOptimization #CodingStandards #MinimalSyntax #AI #DevoutGrowth

23

1 Comment