LLM output structures have become standardized enough that you can now "trick" your tools to use different models, including local ones. Great tip from Ronnie Sheer:

The Hidden Problem with Scaling LLMs on Kubernetes? Most teams scale LLMs the same way: → Add more GPUs → Hope latency stays within limits It “works”… but it’s expensive, inefficient, and hard to operate in shared cloud clusters. Why? Because Prefill and Decode are treated as one big workload even though they behave very differently. The Fix? Split the Work, Not Just the GPUs This is where llm-d comes in. llm-d is a Kubernetes-native way to split LLM serving into prefill and decode, And scale each part independently, without breaking cloud or platform workflows. What llm-d changes? →  Prefill and decode run as separate services →  Each scales based on own latency and throughput needs →  Works in multi-tenant Kubernetes clusters →  Supports different GPU types for different workloads →  Uses standard K8s tools for routing, autoscaling, and failures llm-d makes prefill–decode separation practical for real production systems, not just papers and prototypes. This is where LLM infrastructure is heading. ------------------------------------------------------ #LLM #vLLM #llmd #GPU #AIInfrastructure #MLOps #Inference #AIEngineering #GPUComputing #AIInference #MachineLearning #DeepLearning #MLOps #AIEngineering #GPU #AI #ML #AIPlatforms #ModelServing #K8s #Kubernetes

🚀 🚀 Impressive engineering innovation by the Morph team 🚀 Morph trained its Fast Apply model on Amazon Elastic Compute Cloud (Amazon EC2) P5 instances and deployed it on Amazon SageMaker AI, showcasing a full-stack AI workflow on AWS. They also made the models available via AWS Marketplace, enabling customers to easily discover and deploy the solution. What stands out is the custom inference engine built specifically for code-merging tasks, leveraging speculative decoding to push performance to extraordinary levels. “We trained all these models on AWS… starting at ~1,000 tokens/sec, then 2,000, 4,000, and ultimately reaching 10,000,” said Tejas Bhakta (Founder Morph).

Ok! Second video in my series on #Kubernetes for #AI is up! This one picks up where the last one left off, with a pass-through GPU device, but no Kubernetes cluster yet. So now we get to attach that GPU to a node in the cluster! 🥳 I used #OpenTofu to stand up a bunch of VMs in #Proxmox and wired them together with #K3s. There were some gotchas to be aware of if you're going to use K3s for AI workloads, and I cover those in details. Hint - it has to do with containerd 🫣 This is the last video that's focused entirely on my home lab. Because once we have a cluster stood up, we're in the cloud-native world of working with the Kubernetes API. From here on out, everything I do is applicable across cloud environments...which I'll prove in the third video. 😁 Link is in the comments.

Lately I’ve been reading more than coding. Found this old paper on “scale limits” in machine learning,turns out even if we throw infinite compute at LLMs, there’s still physics and math saying “nope” Reminds me: we chase speed but maybe the future isn’t bigger models, it’s smarter ones. Smaller, cheaper, local. Like the app I built last month for tracking focus runs on a cheap laptop, no cloud, just works. Anyone else quietly rethinking the “bigger is better” trap?

Pretty excited to step up training from my local machine to the GCP cloud to accelerate results for this project! ● The build is creating a photo3d-eval container image — it packages evaluate.py with PyTorch + CUDA into a GPU-enabled Docker image so you can run the model evaluation on a GCE VM with an L4 GPU, pulling checkpoints and training data from GCS.  Digital production pipelines have a LONG way to go towards being able to faithfully reproduce real-life physical goods. This is a big step in the right direction!

Developer Orchestrates Enterprise AI on RTX 3080, Slashing Cloud Costs with Custom Architecture 📌 A developer has built a full-featured AI workflow system, Resilient Workflow Sentinel, that runs entirely on a single RTX 3080-shaving cloud costs by 90% and eliminating data privacy risks. Using a quantized 7B model, it handles real-time task routing, failure alerts, and large-context processing without relying on expensive cloud infrastructure or external APIs. This proves enterprise AI orchestration can be local, private, and affordable-even on consumer hardware. 🔗 Read more: https://lnkd.in/dtgJ343g #Qwen257b #Python #Nicegui #Vram10gb

Fuzzball introducing preview support for provisioning GPU compute on CoreWeave. Orchestrate GPU-intensive AI workflows across on-prem, AWS, and CoreWeave, using the same workflow definitions. Burst to CoreWeave when GPU density is scarce, keep everything unified, and run where the accelerators are. Read the blog post to see how hybrid, multi-cloud GPU orchestration actually works in practice: https://bit.ly/4qfFz1d #AIInfrastructure #MLOps #CloudComputing #HybridCloud #HPC #Fuzzball

Just sharing something I’m realizing these days while learning systems. A lot of concepts in Computer Science are actually simple. What often makes them confusing is the terminology. The same core idea shows up across different layers and tools, just with different names. For example: At the kernel level, we call it namespaces In Docker, it becomes a container In Kubernetes, it’s referred to as a pod In cloud platforms, it’s often called a workload(at some level) The names change, but the underlying concept stays the same — isolation, control, and separation of responsibility. What I’m learning is that instead of memorizing terms, it helps more to focus on: What problem is this solving? What responsibility does this component have? Once the concept is clear, mapping it across technologies becomes much easier. Concepts are stable. Terminology evolves. #ComputerScience #SoftwareEngineering #DevOps #Containers #Docker #Kubernetes #SystemDesign #LearningJourney #EngineeringMindset

Free Resource 🪙 If you have a low-end PC but want to work with high-end GPUs like T4, L4 (up to ~24 GB VRAM), Lightning AI offers access to these resources on its free tier for ML and AI workloads. You can: Use their VS Code–like cloud IDE, or Connect the cloud machine to your local IDE and operate powerful GPUs directly from your own system. A solid option for experimenting with ML models, training, inference, and prototyping without expensive hardware. Explore it at- lightning.ai. good day. #MachineLearning #DeepLearning #AI #GPU #CloudComputing #LightningAI #FreeResources #DataScience #MLOps #Developers