Build KEDA External Scaler with NVML Metrics

A new chapter for our team and a new layer in the AI infrastructure stack: MinIO MemKV. As inference workloads scale, context is a constraint, increasing cost and limiting throughput. MemKV solves this and turns context into a shared resource across GPUs.

IN AI INFERENCE, "Late" IS THE SAME AS "Broken."! If you are waiting for a "Scale Event" to trigger after your latency spikes, you’ve already lost the user. Traditional reactive autoscaling works for web apps, but it fails for LLMs. When memory competition hits, latency doesn't crawl—it explodes. 🎇 SwarmOne provides a different path: 🆕 SLO-Driven Autoscaling. 🆕 Proactive: We scale based on real-time P99 trends, not stale CPU metrics. 🆕 Fast: Adjustments happen in milliseconds, not minutes. 🆕 Efficient: We turn "hoarded" hardware into 90%+ active utilization. How does your team deal with the latency issue? 🧐 At what cost? 💰 #Inference #Scalability #AI #SwarmOne #Performance

AI ambitions are only as strong as the infrastructure behind them. But when GPU costs keep rising and utilization stays invisible, it's hard to know whether you're investing wisely or just spending. Datadog GPU Monitoring, now generally available, gives teams end-to-end visibility, from GPU health and performance to the workloads and costs tied to every piece of hardware. The result: more efficient capacity usage and a clearer path to AI revenue.

Pushing an 8×H100 cluster on Swift Compute and the numbers are looking strong 💪 - 97.4% GPU utilization across all 8 GPUs - 148,162 tokens/sec - Running stable for 4h+ at 72°C This is exactly the kind of efficient, high-throughput performance we’re building for. Still very early (pre-product), but the foundation is coming together nicely. Excited to open this up to more users soon. If you’re training or inferencing at scale and want early access, let us know 👇 #AIInfrastructure #GPUCluster #LLM #SwiftCompute

Deploy Production-Ready vLLM Inference Servers on Kubernetes Using AMD Instinct GPUs 📌 Unlock massive throughput for enterprise LLMs by deploying production-ready vLLM inference servers on Kubernetes using AMD Instinct GPUs. This new framework automates high-performance orchestration, leveraging the MI300X's vast memory to eliminate VRAM bottlenecks and scale workloads seamlessly. It provides a robust, automated path to running large-scale models with high availability and near-zero memory waste. 🔗 Read more: https://lnkd.in/dguRSzVk #Vllm #Kubernetes #Amdinstinct #Gpuoperator #Largelanguagemodels

Shared GPU rental means competing for VRAM, bandwidth, and compute with tenants you don't control. Dedicated bare metal means one server, one tenant, full output, every run. The performance difference is real. At scale, the cost difference usually surprises people. The 8x RTX Pro 6000 Blackwell Max-Q is available now on 1Legion: 768 GB total VRAM, from $1.34/GPU/hr, no egress fees. Talk to an Engineer: https://lnkd.in/dDiWdTqT #baremetalgpu #gpuinfrastructure #generativeai #aiinference

Thanks for sharing Sara Carroll . Weka is positioned extremely well for those customers who've invested in Kubernetes, or plan to. The intersection of containers and AI is here now. The best part, in a hyperscaler, on-premises or both- Weka is the here to help.

This tutorial teaches how to run a production-ready vLLM inference server on Kubernetes with AMD Instinct GPUs using containerd, the AMD GPU Operator, persistent storage, and MetalLB. More: https://ku.bz/D6PyJ-sWz

💡 What does it take to move an #AIFactory from proof-of-concept to production? NVIDIA Enterprise Reference Architectures provide infrastructure guidance for on-premises deployments, defining how #compute, networking, storage, software, and system components integrate into a production-ready #AI platform. Read about three configurations for AI factories, built around NVIDIA #RTX PRO Blackwell Server Edition GPUs,# HGX systems, and GB300 NVL72. ➡️ https://bit.ly/42EC4bc #GPU #DataCenter

Scaling GPUs breaks the network before compute! What I mean by that is: compute seems to be scaling linearly in the upcoming systems, but Network scaling hits: IO limits, switch radix and packaging limits!