Streamlining AI Experiment Setup Using Kubernetes

𝗦𝗲𝗹𝗳-𝗵𝗼𝘀𝘁 𝗖𝗹𝗮𝘂𝗱𝗲 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁�� 𝗮𝘁 𝘀𝗰𝗮𝗹𝗲 — 𝗶𝗻𝘀𝗶𝗱𝗲 𝘆𝗼𝘂𝗿 𝗼𝘄𝗻 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀, 𝘄𝗶𝘁𝗵 𝗮𝗰𝗰𝗲𝘀𝘀 𝘁𝗼 𝘆𝗼𝘂𝗿 𝗼𝘄𝗻 𝘀𝘆𝘀𝘁𝗲𝗺𝘀. Meet 𝗸𝗼𝗺𝗽𝘂𝘁𝗲𝗿-𝗮𝗶 — an open-source, Kubernetes-native platform for running persistent Claude AI agents on your own infrastructure, inside your own private network, with direct access to your internal tools and data. Every agent is a first-class Kubernetes resource (a CRD). That means YAML manifests, kubectl, RBAC, namespaces, Helm — all the tools your team already uses for production workloads now apply to your AI agents. 𝗧𝘄𝗼 𝘄𝗮𝘆𝘀 𝘁𝗼 𝘂𝘀𝗲 𝗶𝘁 👇 𝟭. 𝗥𝘂𝗻 𝗖𝗹𝗮𝘂𝗱𝗲 𝗮𝗴𝗲𝗻𝘁𝘀 𝗶𝗻𝘀𝗶𝗱𝗲 𝘆𝗼𝘂𝗿 𝗶𝗻𝗳𝗿𝗮 — no third-party SaaS, no data leaving your cluster. Agents can reach your internal databases, APIs, Git repos, and MCP connectors (Slack, GitHub, Atlassian, Notion, Google Workspace) — because they live on the same network. 𝟮. 𝗕𝘂𝗶𝗹𝗱 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗼𝗻 𝘁𝗼𝗽 — if your platform or app needs to run agents in the backend, don't reinvent the orchestration layer. Wrap komputer-ai with a simple SDK/API/CLI call and focus on your UX, auth, and business logic instead. 𝗪𝗵𝗮𝘁 𝘆𝗼𝘂 𝗴𝗲𝘁 👇 ✅ 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗲𝗻𝘁 𝗮𝗴𝗲𝗻𝘁𝘀 with their own pod + workspace PVC — survives restarts, sleep cycles, and re-tasks ✅ 𝗠𝗮𝗻𝗮𝗴𝗲𝗿/𝘄𝗼𝗿𝗸𝗲𝗿 𝗼𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 — managers spawn and coordinate sub-agents autonomously ✅ 𝗦𝘁𝗲𝗲𝗿 𝗺𝗶𝗱-𝘁𝗮𝘀𝗸 — redirect a running agent without restarting it ✅ 𝗖𝗿𝗼𝗻 𝘀𝗰𝗵𝗲𝗱𝘂𝗹𝗶𝗻𝗴, 𝘀𝗹𝗲𝗲𝗽𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀, 𝗮𝘂𝘁𝗼-𝗱𝗲𝗹𝗲𝘁𝗲 — full lifecycle control ✅ 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀 streamed over WebSocket — build chat UIs, dashboards, Slack bots in minutes ✅ 𝗖𝗼𝘀𝘁 𝘁𝗿𝗮𝗰𝗸𝗶𝗻𝗴 per task, per agent, per namespace ✅ 𝗦𝗗𝗞𝘀 for Python, Go, and TypeScript — plus a CLI and web dashboard 𝟯 𝘀𝘁𝗲𝗽𝘀 𝘁𝗼 𝗴𝗲𝘁 𝘀𝘁𝗮𝗿𝘁𝗲𝗱 👇 📌 helm install komputer-ai oci://https://lnkd.in/gZpJnWap 📌 kubectl apply -f agent.yaml — or use the CLI, UI, or any SDK 📌 Stream events in real time as your agent works 🌐 https://lnkd.in/gssGTm68 — star it, fork it, build on it.

This GitHub repo is a gold mine for DevOps and Platform Engineers building AI Agents. Most people are still running agents like toys: - chat window - local script - random MCP config - no RBAC - no logs - no rollback - no real deployment model That is fine for demos. But if you want agents in production, the question changes. Not: "Which model should I use?" But: "Where does this agent run?" "Who can approve its actions?" "How do I observe every tool call?" "How do I roll it back?" "How do I version its behavior?" "How do I stop it from touching the wrong system?" This is why I liked `kagent`. It is a Kubernetes-native framework for building, deploying, and managing AI agents. Repo: kagent-dev/kagent Stars: 2.7k+ Forks: 543+ Commits: 1,232+ Latest release: v0.9.2 What it gives you: 1. Agents as Kubernetes resources You define agents using Kubernetes custom resources. That means your AI agent can move through the same workflow your infra already uses: - YAML - Git - PR review - kubectl - GitOps - CI/CD - rollout - rollback This is a big shift. Agents stop being random scripts and start becoming managed workloads. 2. MCP tools for real cloud-native systems kagent comes with MCP tools for: - Kubernetes - Istio - Helm - Argo - Prometheus - Grafana - Cilium - and more This matters because agents are only useful when they can touch real systems safely. A DevOps agent that cannot inspect a cluster, read metrics, check deployments, or understand services is just a chatbot with better branding. 3. ToolServers as reusable primitives Tools are represented as Kubernetes custom resources called ToolServers. Multiple agents can reuse the same tool server. That is a very platform-engineering way to think about agents. Instead of every team wiring its own tool access, you expose approved capabilities once and reuse them safely. 4. Observability is part of the model kagent supports OpenTelemetry tracing. For AI agents, this is not optional. You need to know: - what prompt ran - what tool was called - what data came back - what action was taken - where the failure happened - how much the agent actually did Without traces, logs, and auditability, nobody serious will trust agents with production systems. 5. It fits how platform teams already work The best part is not that it is "AI". The best part is that it uses primitives DevOps teams already understand: - Kubernetes - CRDs - RBAC - GitOps - observability - service mesh - MCP - declarative config That is the right direction. AI agents will not become production systems because prompts get longer. They will become production systems when they get the same operating layer we gave every other workload. If you are a DevOps, SRE, platform, or cloud-native engineer, bookmark this repo. This is one of those projects that shows where the agent infrastructure layer is going. Repo: https://lnkd.in/ee2UwJ5Z

Deploy HPC Infrastructure and AI workloads with Kubernetes: Since I've been at OCI, I've had unique insights into some of the challenges customers face onboarding onto CSPs. One of those challenges is onboarding the large catalogue, which is further complicated by the infra required to run AI workloads like LLM inference and fine-tuning. Because of this, my team developed an API layer that sits on top of the Kubernetes control plane to provide an easy interface to deploy containerized applications to OCI compute. Currently, we provide a wide range of support for inference, with distributed inference, multi-instance GPU, and pod and node autoscaling support for GPU accelerated workloads. In addition to this, we also provide custom transformers fine-tuning, or the ability to benchmark specific GPU offerings with the mlperf fine-tuning benchmark. We also bake in application monitoring with Prometheus and Grafana, and enable autoscaling by leveraging inference application metrics such as time-to-first-token, end-to-end latency, or whatever is required. Additionally, we deploy a convenient portal for people who are less interested in interacting with these resources through code as a point and click solution. This is all enabled via terraform stacks, which require minimal overhead for deployment of resources such as policies, networking, database, and helm charts. I'm excited to announce this because I think the team has done a great job simplifying a complex deployment process for a lot of teams. Check out the blog: https://lnkd.in/e6kzUabD #softwareengineering #gpus

Published a Hands-On End-to-End series - Deploying ML Models on Kubernetes with Auto-Scaling (HPA) & Monitoring (Prometheus) 👩🏻🏫 📚 Topics Covered [1] Introduction & Project Setup – Understand the project scope and architecture [2] Setting Up Podman & Kind – Install and configure tools for local Kubernetes cluster management [3] Creating a Kubernetes Cluster – Initialize and configure your Kubernetes environment [4] Deploying Persistent Storage – Configure Persistent Volumes and Claims for storing model data [5] Setting Up ConfigMap – Manage application settings with Kubernetes ConfigMaps [6] Deploying the ML Application – Containerize and deploy the ML model API on Kubernetes [7] Exposing the Service & Auto-Scaling – Make the service accessible and configure Horizontal Pod Autoscaler (HPA) [8] Setting Up Prometheus for Monitoring – Integrate Prometheus to monitor system and application metrics [9] Testing the API & Metrics – Validate the deployed API and monitor Prometheus outputs [10] Debugging & Troubleshooting – Identify and resolve common issues in production deployments https://lnkd.in/ehpzj8yG Happy learning and happy deploying 🚀 #MLOps #Kubernetes #MachineLearning

If you’re building AI agents that need to work reliably in production, not just in demos, this is the full-stack setup I’ve found useful From routing to memory, planning to monitoring, here’s how the stack breaks down 👇 🧠 Agent Orchestration → Agent Router handles load balancing using consistent hashing, so tasks always go to the right agent → Task Planner uses HTN (Hierarchical Task Network) and MCTS to break big problems into smaller ones and optimize execution order → Memory Manager stores both episodic and semantic memory, with vector search to retrieve relevant past experiences → Tool Registry keeps track of what tools the agent can use and runs them in sandboxed environments with schema validation ⚙️ Agent Runtime → LLM Engine runs models with optimizations like FP8 quantization, speculative decoding (which speeds things up), and key-value caching → Function Calls are run asynchronously, with retry logic and schema validation to prevent invalid requests → Vector Store supports hybrid retrieval using ChromaDB and Qdrant, plus FAISS for fast similarity search → State Management lets agents recover from failures by saving checkpoints in Redis or S3 🧱 Infrastructure → Kubernetes auto-scales agents based on usage, including GPU-aware scheduling → Monitoring uses OpenTelemetry, Prometheus, and Grafana to track what agents are doing and detect anomalies → Message Queue (Kafka + Redis Streams) helps route tasks with prioritization and fallback handling → Storage uses PostgreSQL for metadata and S3 for storing large data, with encryption and backups enabled 🔁 Execution Flow Every agent follows this basic loop → Reason (analyze the context) → Act (use the right tool or function) → Observe (check the result) → Reflect (store it in memory for next time) Why this matters → Without a good memory system, agents forget everything between steps → Without planning, tasks get run in the wrong order, or not at all → Without proper observability, you can’t tell what’s working or why it failed → And without the right infrastructure, the whole thing breaks when usage scales If you’re building something similar, would love to hear how you’re thinking about memory, planning, or runtime optimization 〰️〰️〰️〰️ ♻️ Repost this so other AI Engineers can see it! 🔔Follow me (Aishwarya Srinivasan) for more AI insights, news, and educational resources 📙I write long-form technical blogs on substack, if you'd like deeper dives: https://lnkd.in/dpBNr6Jg