Streamlining LLM Inference for Lightweight Deployments

Transformer-Lite High-efficiency Deployment of Large Language Models on Mobile Phone GPUs The Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite's performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU-based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.

If you’re an AI engineer trying to optimize your LLMs for inference, here’s a quick guide for you 👇 Efficient inference isn’t just about faster hardware, it’s a multi-layered design problem. From how you compress prompts to how your memory is managed across GPUs, everything impacts latency, throughput, and cost. Here’s a structured taxonomy of inference-time optimizations for LLMs: 1. Data-Level Optimization Reduce redundant tokens and unnecessary output computation. → Input Compression:  - Prompt Pruning, remove irrelevant history or system tokens  - Prompt Summarization, use model-generated summaries as input  - Soft Prompt Compression, encode static context using embeddings  - RAG, replace long prompts with retrieved documents plus compact queries → Output Organization:  - Pre-structure output to reduce decoding time and minimize sampling steps 2. Model-Level Optimization (a) Efficient Structure Design → Efficient FFN Design, use gated or sparsely-activated FFNs (e.g., SwiGLU) → Efficient Attention, FlashAttention, linear attention, or sliding window for long context → Transformer Alternates, e.g., Mamba, Reformer for memory-efficient decoding → Multi/Group-Query Attention, share keys/values across heads to reduce KV cache size → Low-Complexity Attention, replace full softmax with approximations (e.g., Linformer) (b) Model Compression → Quantization:  - Post-Training, no retraining needed  - Quantization-Aware Training, better accuracy, especially <8-bit → Sparsification:  - Weight Pruning, Sparse Attention → Structure Optimization:  - Neural Architecture Search, Structure Factorization → Knowledge Distillation:  - White-box, student learns internal states  - Black-box, student mimics output logits → Dynamic Inference, adaptive early exits or skipping blocks based on input complexity 3. System-Level Optimization (a) Inference Engine → Graph & Operator Optimization, use ONNX, TensorRT, BetterTransformer for op fusion → Speculative Decoding, use a smaller model to draft tokens, validate with full model → Memory Management, KV cache reuse, paging strategies (e.g., PagedAttention in vLLM) (b) Serving System → Batching, group requests with similar lengths for throughput gains → Scheduling, token-level preemption (e.g., TGI, vLLM schedulers) → Distributed Systems, use tensor, pipeline, or model parallelism to scale across GPUs My Two Cents 🫰 → Always benchmark end-to-end latency, not just token decode speed → For production, 8-bit or 4-bit quantized models with MQA and PagedAttention give the best price/performance → If using long context (>64k), consider sliding attention plus RAG, not full dense memory → Use speculative decoding and batching for chat applications with high concurrency → LLM inference is a systems problem. Optimizing it requires thinking holistically, from tokens to tensors to threads. Image inspo: A Survey on Efficient Inference for Large Language Models ---- Follow me (Aishwarya Srinivasan) for more AI insights!

What if your LLM could reuse work and respond 5-10× faster? That’s exactly what LMCache delivers. What is LMCache? It’s the open-source “KV cache layer” for LLMs — designed to store and reuse key/value caches across queries, sessions and even engines. Built for high-volume, long-context systems. Evaluations show up to 15× throughput improvements when paired with engines like vLLM. Why This Matters Right Now Latency kills UX. Every extra millisecond waits hit adoption. LMCache slashes response time by re-using caches. GPU cycles cost money. Re-computation means wasted resources. LMCache allows reuse across workloads, reducing GPU load. Context & multi-round workflows are exploding. RAG systems, agent pipelines, conversational contexts — LMCache fits them all. It’s production-ready and open-source. No black-box: you can inspect, integrate, extend. Typical Use Cases: -Agentic systems that make multi-turn decisions -RAG pipelines that reuse retrievalable contexts -Long-form applications (document processing + summarization) -Multi-engine inference clusters / cloud-scale deployments Plug into your engine and enable KV-cache reuse across queries & threads. If you’re building LLM-based systems for scale, this isn’t one more library — it’s a fundamental architecture upgrade. Mark this: The future of LLM inference isn’t just bigger models — it’s smarter reuse.

70% Smaller LLMs With Zero Accuracy Loss: Introducing DFloat11 Compression ... 👉 Why This Matters Large language models are hitting hardware limits: - Lossy quantization (8-bit/4-bit) reduces model size but alters outputs, risking accuracy drops in reasoning, coding, and niche tasks - Traditional lossless compression works for storage but fails during GPU inference due to serial decoding bottlenecks 👉 What Changed The DFloat11 framework achieves: - 30% size reduction for models like Llama-3, Qwen, and Gemma - Bit-for-bit identical outputs compared to original BFloat16 models - Efficient GPU inference via parallel decompression, avoiding CPU offloading delays The Core Insight: BFloat16’s exponent values are highly repetitive. By applying entropy coding (shorter codes for frequent patterns), DFloat11 compresses exponents while keeping signs/mantissas intact. 👉 Technical Breakthroughs 1️⃣ GPU-friendly decompression: - Splits large lookup tables into SRAM-sized chunks for fast access - Coordinates 1,000s of threads to decode variable-length codes in parallel 2️⃣ Transformer-block-level processing: - Batches weight decompression to maximize GPU utilization - Adds minimal latency (amortized over large batches) 👉 Real-World Impact - 1.9–38.8x faster than CPU-offloaded inference - Enables 5.3–13x longer context windows by freeing GPU memory - Runs 810GB models (e.g., Llama-3.1-405B) on 8x80GB GPUs – previously impossible Validation: - Identical accuracy on MMLU, TruthfulQA, and perplexity benchmarks - 100% weight reconstruction accuracy post-decompression 👉 Why It’s a Big Deal DFloat11 removes the “compromise mindset” in LLM deployment. Engineers no longer need to choose between model size, accuracy, and hardware costs – all three improve simultaneously.

No one really explains how llama.cpp works under the hood. For deploying LLMs on Edge or CPU, most guides stop at “use llama.cpp”, but they don't explain what’s happening under the hood. ✅ So I decided to fix that. I spent hours digging through the codebase, PRs, and community threads, and turned it all into a single, clear sequence diagram showing how it really works. My goal was to see what's happening, to understand each component, from loading up an LLM Checkpoint, up to generating the first token. Why is this important? 1️⃣ Frontier LLMs are built for high-compute environments. 2️⃣ But small language models (SLMs) are catching up, some even matching larger LLMs on key tasks. This means that with the appropriate toolkit, anyone could optimize and run them locally on their consumer Hardware, CPUs, or GPUs, and Edge devices. Having your own GPT-5 level LLM running on a CPU is impossible. But running Gemma 3, Llama 3.2, Phi-4, or Nemotron (3B–12B) is totally doable. In this deep dive, I cover: > GGML - the ML Tensor Library and how it parses LLM checkpoints. > GGUF - the format for storing quantized LLM models and Quantization types. > The high-level architecture of how everything fits together. > Source code overlays and sequence diagrams. Key points to know: 1/ llama.cpp is a pure C++ inference engine for LLMs, cross-platform (x64, ARM64, x86) 2/ GGML + GGUF + llama.cpp form a complete, deployable edge stack 3/ You can run modern LLMs with minimal dependencies and full control. 📌 Find the deep dive link in the first comment. It’s everything you need to understand the stack, not just use it. Enjoy!

🚀 𝗜𝗻𝘀𝗶𝗱𝗲 vLLM: 𝘄𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝗶𝘁 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗯𝗲𝘀𝘁 𝗳𝗼𝗿 𝗟𝗟𝗠 𝘀𝗲𝗿𝘃𝗶𝗻𝗴 vLLM Is my favorite inference engine for self-hosting LLMs. It feels snappier because its design keeps GPUs busy and memory tidy. Here are the parts that matter when you’re shipping real apps. 🔩 𝗖𝗼𝗿𝗲 𝗲𝗻𝗴𝗶𝗻𝗲 𝗶𝗱𝗲𝗮𝘀  • 𝗣𝗮𝗴𝗲𝗱𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 treats the KV cache like virtual memory: fixed-size pages that can be allocated, compacted, and reused—less copying/fragmentation and higher GPU utilization under bursty traffic.  • 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗯𝗮𝘁𝗰𝗵𝗶𝗻𝗴 admits new requests at token boundaries so GPUs don’t idle for the slowest prompt; throughput rises without hurting p50/p95 latency.  • 𝗣𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 shares overlapping headers (system prompts, RAG/tool preambles) to cut repeat compute and speed time-to-first-token.  • 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗸𝗲𝗿𝗻𝗲𝗹𝘀 & graphs reduce launch overhead; prefill/decode paths are tuned for chats and long contexts. 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻  • Tensor & pipeline parallelism split weights/layers across GPUs so larger models fit and tokens stay in lockstep.  • Multi-node scheduling preserves batching/paging across machines—scale out without giving up efficiency.  • One-model-per-process keeps blast radius small; run many vLLM servers and route via a gateway. 🧰 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿-𝗳𝗿𝗶𝗲𝗻𝗱𝗹𝘆 𝘀𝗲𝗿𝘃𝗶𝗻𝗴  • 𝗢𝗽𝗲𝗻𝗔𝗜-𝘀𝘁𝘆𝗹𝗲 𝗲𝗻𝗱𝗽𝗼𝗶𝗻𝘁𝘀 (chat/completions/embeddings) ease migrations.  • 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗯𝘂𝗳𝗳𝗲𝘁 (INT8/INT4, GPTQ/AWQ/AutoRound, FP8) trades tiny quality for big cost/latency wins.  • 𝗖𝗿𝗼𝘀𝘀-𝘃𝗲𝗻𝗱𝗼𝗿 𝗯𝗮𝗰𝗸𝗲𝗻𝗱𝘀 keep options open across accelerators and clouds.  • Streaming first with SSE for faster perceived latency. 💡 𝗪𝗵𝘆 𝗶𝘁 𝗺𝗮𝘁𝘁𝗲𝗿𝘀  • Lower $/token via better GPU saturation.  • Tighter tail latency keeps SLOs green.  • Operational simplicity—paging, caching, batching reduce custom CUDA and brittle schedulers. ⚙️ 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝘁𝗶𝗽𝘀 • Keep prompts DRY so prefix caching hits often. • Use shorter max_tokens + streaming; request more if needed. • Right-size KV blocks and batch sizes to traffic shape. • Measure prefill vs decode throughput; long contexts are often prefill-bound. 🧪 𝗪𝗵𝗲𝗿𝗲 𝘃𝗟𝗟𝗠 𝘀𝗵𝗶𝗻𝗲𝘀 • Agent platforms with many short turns. • RAG APIs with shared system prompts. • Consumer chat with unpredictable spikes. • Enterprise multi-tenant backends needing strong isolation. 🔮 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆 vLLM’s speed comes from the combo of paged KV memory, continuous batching, smart caching, and lean kernels—turning GPUs into well-fed token factories with speed, cost control, and predictability. Aleksa Gordić’s deep-dive blog is the clearest explanation of the vLLM engine I’ve seen 👉 https://lnkd.in/gRgiC_45  🔗 #vLLM #LLM #SelfHosting #AIInfrastructure #Inference #GPU #CUDA #SystemsDesign #AIAgents #Latency #Throughput #Quantization #KVCache #PagedAttention

Why You Should Consider llm-d for Your LLM Workloads At IBM Research, we're constantly evaluating the next-generation tools that can make AI inference both faster and more cost-effective. llm-d stands out for several reasons: 1. Disaggregated Inference - By separating the heavy "prefill" phase from the latency-sensitive "decode" phase, llm-d lets each step run on the most appropriate hardware, boosting GPU utilization and cutting expenses. 2. Smart Caching & KV-store Reuse - Repeated prompts or multi-turn conversations reuse previously computed tokens, delivering noticeable latency reductions for RAG, agentic workflows, and long-context applications. 3. Kubernetes-native Scaling - The platform integrates with the Kubernetes Gateway API and vLLM, enabling automatic load balancing based on real-time metrics (GPU load, memory pressure, cache state). This makes it easy to expand from a single node to a full cluster without re-architecting your services. 4. Open-source and Enterprise-grade - Backed by a community that includes Red Hat, NVIDIA, Google, and IBM, llm-d benefits from rapid innovation while remaining transparent and production-ready. 5. Designed for Modern AI Use Cases - Whether you're building retrieval-augmented generation pipelines, long-running conversational agents, or any workload that demands high throughput and low latency, llm-d provides the performance foundation you need. If you're looking for a solution that maximizes hardware efficiency, reduces operating cost, and scales seamlessly in a cloud-native environment, give llm-d a closer look. Main page: https://llm-d.ai Your turn: Have you tried llm-d or a similar distributed inference framework? What challenges are you facing with large-model serving, and how are you addressing them? I’d love to hear your experiences and insights.

🚀 What if your LLM got faster the more you used it? - That is the promise of self-adaptive speculative decoding! Left = Slow LLM, Middle = Fast LLM, Right = Faster LLM adapting to your prompts over time! Red slow tokens are from 670B param DeepSeek, blue/black fast ones are from a 8B small LLM trained to copy the big model! Speculative decoding lets a small model, a speculator, “guess ahead” what a large model will say, and the big model only needs to verify, not regenerate, every token. The result: 2-3× faster inference for large LLMs. But there’s been a catch: traditional speculators are static, fixed after training, unable to adapt when workloads or domains shift. This means that over time they get worse at guessing the larger model tokens as data distributions shift. The promise behind Together AI’s new AdapTive-LeArning Speculator (ATLAS) System is that the small model can learn and adapt to your prompts so that it’s correct more often and can speed the larger model up even if the questions you're asking your model change over time. 🧠How does it work? ATLAS introduces: 🔹 A static speculator trained on broad data for stability and fallback. 🔹 A lightweight adaptive speculator that continuously fine-tunes on real-time traffic, learning from new inputs as they arrive. 🔹 A confidence-aware controller that dynamically balances accuracy and speed by adjusting speculation lookahead. 📈 The results? 🔹 Up to 400% faster decoding (from 105 to 501 tokens/sec). 🔹 Sustained gains during RL training, cutting rollout time by >60%. 🔹 Real-time specialization to each user’s evolving input patterns. In short: the more you use it, the faster it gets! 🚀 Full Blog below 👇

You are paying for billions of tokens each day before generating a single useful output 💸 At Twitter, we cut ads ranking prediction costs by 85% - not with a better model, but by fixing payload bloat. The same pattern is showing up again with MCP. It’s brilliant for developer workflows, but naive production deployments create a “context-window tax” that compounds silently. Here's the math people aren't doing: → ~3,000 tokens of tool/schema context per request → 500k daily requests → billions of tokens/day Yes, caching helps - a lot. But only if prompts are structured for reuse. Most aren’t. Here are the top 4 things to solve this architecture problem: ❶ Default to cheap routers. Regex, embeddings, small fine-tuned models, or at most Flash/Haiku/nano-tier LLMs. Frontier models should be the last resort. The cost delta is 3–5x with negligible routing quality difference! ❷ Decouple orchestration from reasoning. Lightweight models handle tool use & APIs. Frontier models handle synthesis, multi-step reasoning, and ambiguity. Don’t use a sledgehammer to sort mail. ❸ Treat context like a production resource. Don’t inject every tool schema into every request. Scope tools, compress schemas, and load lazily. Every token costs on every call. ❹ Cache aggressively, but correctly. Prompt caching can cut costs up to 90% (Anthropic, OpenAI, Google DeepMind). But it only works if prefixes are stable and prompts are reusable. The best ML systems aren't the most clever. They're the ones that minimize tokens, isolate expensive reasoning, and make cost-quality tradeoffs explicit. This is Part 1 of my MCP production teardown. Over the next few weeks, I’ll share insights on Shadow AI protocols, model-agnosticism, memory vs reflex, and more. If you're building Gen AI systems at scale, I’d love to hear from you. Curious what’s been your highest cost or latency bottleneck so far.

For developers and platform engineers managing LLM infrastructure, the llm-d team just dropped a deep dive into solving one of the hardest problems in inference: Load Balancing requests for LLMs. The standard approach uses heuristic weights (queue depth, memory pressure, cache locality). But in production, these signals conflict, and manual tuning can't keep up with bursty traffic. The solution? Predictive-Latency Based Scheduling. Instead of guessing, a lightweight XGBoost model is used which is trained from live traffic. The model predicts:  🔹 TTFT (Time to First Token)  🔹 TPOT (Time Per Output Token) The results are massive:  43% improvement in P50 end-to-end latency.  70% improvement in TTFT. It dynamically balances "spreading" (to reduce batch size) vs. "consolidation" (to maximize KV cache reuse) based on real-time performance, not static guesses. Check out the full breakdown of how they built it and the benchmark results: 🔗 https://lnkd.in/gwdB-kV9 #LLM #GenerativeAI #MLOps #Kubernetes #AIInfrastructure #LLMInference