Jon Stanga

Terraform's original sin is the state file. We built modern infrastructure on fragile JSON blobs in S3. Every project its own island, no global view, no history, just grep and jq when you need answers. Here's the reality: Terraform sold us "infrastructure as code," but stopped short of making it infrastructure as data. What we got was infrastructure as scattered text files with the relational depth of a Post-it note. State should be a living data source. Queryable. Auditable. Real-time. Exposed through APIs your tooling can actually use. #terraform

501

106 Comments

If you’ve worked with Apache Kafka, you know the trade-offs: Blazing-fast local disk writes but costly replication across zones. Tiered storage for long retention but complexity in managing hot vs cold data. The Kafka community has been asking: Can we make object storage the first-class durable store without losing Kafka’s semantics? That’s where Diskless Topics (KIP-1150 & KIP-1163) by Aiven started the journey — rethinking Kafka’s durability around object storage as the source of truth. Now comes Diskless 2.0 — a unified, zero-copy approach that: ✅ Removes parallel “disk vs tiered” estates ✅ Simplifies the produce/consume path ✅ Enables zero-copy migration (flip a topic from classic to diskless without reprocessing) ✅ Cuts costs by eliminating cross-AZ replication and reducing broker disk reliance This isn’t just an incremental patch. It’s a paradigm shift in how we think about Kafka in the cloud era — where durability and economics favor object storage, and Kafka brokers can finally be “lighter.” Of course, challenges remain: -Latency sensitivity (classic still wins for ultra-low-latency cases) -Compaction and deletion semantics (work in progress) -Operational readiness for large clusters But that’s the beauty of open source — progress through community iteration. And Diskless 2.0 feels like a community win, not just a vendor push. 👉 Dive deeper: Read the KIP-1150 proposal here: fandf.co/3VlQxW1 Explore Aiven’s take on Diskless 2.0: fandf.co/46pczw0 What do you think? Could Diskless 2.0 become the default way to run Kafka in the next 3–5 years, or will classic/tiered still dominate latency-critical workloads? Kudos to Aiven for releasing Diskless 2.0 and sponsoring this post. #Kafka #Sponsored #Aiven

266

49 Comments

The future is here - Kafka is cheap to run. It just isn't evenly distributed yet. Case studies with Diskless Kafka architectures are starting to pop up now. The cost savings are as shocking as marketed: • WarpStream 🤝 Pixel Federation - $60k/yr AWS MSK spend down to <$10k/yr with WarpStream • Aiven 🤝 OpsHelm - $50k/yr AWS MSK spend down to <$10k/yr with Diskless Kafka BYOC Both are 5-6x cost decreases. This is what I mean when I say the Kafka market is seeing massive cost deflation through commoditization and innovation. Watch this space.

168

17 Comments

Platform Engineering exists because we turned Kubernetes into a Frankenstein monster and needed someone to keep it alive. The fun starts when you try to upgrade. Your cluster wants Kubernetes 1.28, but your monitoring stack only works with 1.26. Your service mesh throws a fit every time the API changes. What began as "developers deploy their own stuff" became "we need a team of specialists just to prevent this thing from exploding." And voilà! Platform Engineering was born. 𝗡𝗼𝘁 𝗳𝗿𝗼𝗺 𝘃𝗶𝘀𝗶𝗼𝗻𝗮𝗿𝘆 𝗽𝗹𝗮𝗻𝗻𝗶𝗻𝗴, 𝗯𝘂𝘁 𝗳𝗿𝗼𝗺 𝗽𝘂𝗿𝗲 𝗻𝗲𝗰𝗲𝘀𝘀𝗶𝘁𝘆. 𝗦𝗼𝗺𝗲𝗼𝗻𝗲 𝗵𝗮𝗱 𝘁𝗼 𝗯𝗮𝗯𝘆𝘀𝗶𝘁 𝘁𝗵𝗲 𝗺𝗼𝗻𝘀𝘁𝗲𝗿 𝘄𝗲 𝗰𝗿𝗲𝗮𝘁𝗲𝗱. The job posting practically writes itself: "Wanted: Person to maintain compatibility between 50 different tools while pretending this is all intentional design." Platform Engineers do important work - don't get me wrong. But they exist because we couldn't resist turning a container orchestrator into a Swiss Army knife that cuts everything except what you actually need. We solved deployment complexity by creating platform complexity. Then we hired people to manage that complexity and called it innovation.

349

70 Comments

Kubernetes isn't hard - building dynamic, scalable and stable infrastructure is. Kubernetes isn't hard - Changing legacy people processes and breaking down silos is. Kubernetes isn't hard - learning new patterns is. Kubernetes is not very hard standalone but implementing in an environment that isn't conducive to cloud native patterns can be messy and complicated. Don't let people blow smoke about how it's overly complex - by itself - it isn't. Kubernetes does a great job of of simplifying the management of the complexity - but make no mistake - none of the problems you are solving are by any means "easy". When people start using declarative infrastructure platforms for the first time they don't realize the hard part is front loaded so these decisions never need to come up post deployment. This means it's always harder to deploy then it is to maintain. Once you make the decisions - you never have to make them again. But before you start - You better know exactly what you want. So "I just want to get thing running" is a better pain-point than "I just wish this thing would keep working so I can sleep a full night". The problem isn't that you need to learn Kubernetes. The problem is you need to know your environment and how Kubernetes can (or can't) help.

72

4 Comments

If you are looking for an open-source solution for alerting to multiple channels and on-call. Check this repo. It's a tool that supports alerting across multiple channels with easy custom messaging and on-call integrations. Compatible with any tool supporting webhook alerts, it's designed for modern DevOps teams to quickly respond to production incidents. #SRE #DevOps

79

1 Comment

🚨 If you’re running Kubernetes and haven’t installed Stakater Forecastle yet… You’re missing something incredibly useful. Ever joined a new team and spent your first few days just… asking for URLs? 1️⃣ “Where’s the Grafana dashboard?” 2️⃣ “Do we have ArgoCD running somewhere?” 3️⃣ “Wait… which namespace is that in again?” In most Kubernetes clusters, every team has a dozen services but no one knows where they live. Docs get outdated. Slack threads go missing. You end up bookmarking half the internet just to stay sane. That’s why we built Stakater Forecastle; a simple, clean landing page for your cluster services, auto-discovered from ingress and route metadata. 🧭 Think of it as your internal cluster portal zero config, just drop it in and go. With Forecastle you get: ✅ A centralized view of all exposed apps ✅ Grouping by capability ✅ Click-and-go access for developers, testers, and stakeholders ✅ Instant usability no dashboards to build or configure 🔐 Works on any Kubernetes distro, including OpenShift 🌍 Loved by the community (with 660+ GitHub stars and growing) 💡 It’s one of those tools that feels obvious… after you start using it. 💬 Using Forecastle already? Drop a comment with your use case! Haven’t tried it yet? 👉 Check it out: https://lnkd.in/ekgfSxR #Kubernetes #Stakater #Forecastle #DevEx #DeveloperTools #PlatformEngineering #OpenSourceTools #InternalPortals #Ingress #OpenShift #CloudNative #ClusterUsability

64

13 Comments

Most Kubernetes outages aren’t complex. They’re just dumb. So I built a 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 𝗦𝗮𝗻𝗶𝘁𝘆 𝗣𝗮𝗰𝗸 with 10 𝗼𝗻𝗲-𝗹𝗶𝗻𝗲𝗿𝘀 that instantly catch the dumb stuff before it nukes your app. Here’s your checklist. I run these before every war room. 1. Find crash loops fast kubectl get pods --all-namespaces | grep CrashLoopBackOff 2. What’s spamming your event logs? kubectl get events --sort-by='.lastTimestamp' | tail -n 20 3. App down? Maybe your probe is misfiring. kubectl describe pod <pod-name> | grep -i "probe" 4. Service with no backing pods = silent fail kubectl get endpoints -A | grep -v "ENDPOINTS" 5. Spot stuck deployments kubectl rollout status deployment/<name> --watch 6. Find restarts > 3 (your quiet troublemakers) kubectl get pods --all-namespaces --field-selector=status.phase=Running | awk '$5 > 3' 7. Is the node low on disk or memory? kubectl describe node <node-name> | grep -A5 "Non-terminated Pods:" 8. What killed your pod? kubectl describe pod <pod-name> | grep -A10 "State:" 9. Secret or configmap typo? Check the mounts. kubectl describe pod <pod-name> | grep -i "Volumes:" 10. Most common one: bad image pull. kubectl get pods | grep ImagePullBackOff Run this once a week. Or better, automate it. Most of these catch issues before Prometheus alerts even fire. What’s your go-to one-liner that saved your a** on a Friday night deploy?

110

7 Comments

🔥 Most people using Kubernetes don't actually understand how it works. They write YAML and run kubectl apply, but when things break, they're lost. The secret? Controllers are the beating heart of K8s. This video breaks down how they really work 👇 https://lnkd.in/dYxw2dxg

224

8 Comments

Terraform Meets AI: How LLMs Will Redefine Infrastructure as Code Infrastructure as Code (IaC) has been one of the most transformative practices in modern cloud engineering. Tools like Terraform gave organizations the ability to codify infrastructure, achieve consistency, and automate provisioning at scale. Yet, as powerful as Terraform is, it remains bound to a world of declarative configuration files, modules, and syntax that require deep expertise. Enter Large Language Models (LLMs)—and suddenly, the future of IaC looks very different. Instead of writing HashiCorp Configuration Language (HCL) by hand, imagine expressing intent in natural language: “Deploy a secure, production-ready EKS cluster across three availability zones with autoscaling enabled, monitored by CloudWatch, and integrated with IAM best practices.” An LLM-powered system doesn’t just generate Terraform code—it validates policies, suggests optimizations, and connects with compliance checks before execution. The result is not just automation but intelligence embedded in infrastructure workflows. Here’s how LLMs will redefine Terraform and IaC: 🔹 Intent-Driven Provisioning Engineers and architects will shift from “how to configure” to “what outcome to achieve.” LLMs handle the translation into Terraform modules, dependencies, and variables. 🔹 Policy-Aware by Design No more after-the-fact compliance scanning. LLMs can embed organizational security, cost, and governance policies directly into the Terraform plan generation process. 🔹 Continuous Remediation Infrastructure drift and misconfigurations are inevitable. With LLMs, remediation isn’t reactive—it’s predictive. The system detects anomalies and suggests Terraform fixes automatically. 🔹 Human + AI Collaboration ChatOps integrations will allow teams to review, approve, and modify Terraform plans directly in collaboration tools like Slack or Teams. IaC becomes a dialogue, not just a pull request. 🔹 Multi-Cloud Simplification LLMs can abstract away provider-specific nuances, making multi-cloud IaC orchestration far more intuitive and less error-prone. As a solution architect, I see this as a paradigm shift. Terraform will continue to provide the foundational execution engine, but LLMs will act as the intelligent layer that interprets, validates, and evolves infrastructure strategies in real time. This isn’t about replacing engineers. It’s about amplifying human capability—freeing teams from low-level syntax struggles so they can focus on designing resilient, secure, and business-aligned systems. Terraform gave us automation. LLMs bring us intelligence. Together, they unlock a future where infrastructure is no longer just code—it becomes living, adaptive, and collaborative intelligence.

71

5 Comments

Everybody says not to start with Kubernetes but to paraphrase Kelsey Hightower, "Kubernetes is the new Linux", so it's important to know how to build on it. Here's how I would build a future-proof Kubernetes cluster on AWS and GitHub that will allow you to iterate quickly and scale up when you need to. I like to think of Kubernetes-based infrastructure in three layers. They need to be deployed in this order: 1) Physical infrastructure layer (a set of nodes running Kubernetes) 2) Application infrastructure layer (the services you need to run your apps, things like: cert managers, ingress controllers, etc) 3) Application layer (your actual apps you want to run) Here's the high-level recipe: 1) Start by deploying AWS EKS and its requisite resources with Terraform. This should be a single Terraform module in a GitHub repo with a GitHub Action for automatically running `terraform apply` when you push a commit to main. 2) Use Terraform to bootstrap FluxCD on your cluster to allow you to run "GitOps". This can be in the same module as the above just add the right dependencies. 3) Store your application infrastructure services as Helm charts in a GitHub repo. Use FluxCD HelmRelease resources to automatically sync these charts to your cluster (this is the GitOps part). This should include things like AWS External Secrets Operator, Nginx Ingress Controller, etc. 4) Containerize your app components and add GitHub Actions to their repos to build and publish images to ECR when you commit to main. 5) Write Helm charts for each of your app's components (frontend, backend, queue, DB, etc) and add HelmRelease resources for them. 6) Add ImageUpdateAutomation resources to scan ECR and update your Helm chart values for each of your app components. This will trigger a deployment from the HelmRelease setup we did in step 5. This can be a lot of work and there are lots of details but the end result is: - To deploy your latest code to the cluster all you have to do is commit to the main branch. - To update a Helm chart all you have to do is commit to the main branch and bump the version tag in the HelmRelease resource. - To scale up your apps just edit one value in their HelmRelease. - To scale up your cluster just change one number in the Terraform values. - To debug your system just use standard `kubectl` commands that AI knows well. - All of this infrastructure and configuration is well versioned and centralized in a single repo so you can add branch protections and prevent drift. This might be overkill for your vibecoded todo app but for any system requiring serious infrastructure considerations I'd argue its better to pay this cost upfront before your customers come knocking about downtime and latency issues later.

184

10 Comments

Should you go all-in on Kubernetes, cloud-native and Terraform from Day 1? If you know you'll need to scale infra quickly, 100%. We just rolled out Hatchet Cloud in three new regions (Virginia, Ireland, and Sydney), and it only took us a few days. We have a complex set of requirements which make it difficult to use any sort of PaaS/abstraction on our infra. Instead, our stack is: - Terraform for managing all infrastructure - regions, clusters, load balancers, etc - Cloud-native infrastructure dependencies, excluding Postgres providers (RDS, Ubicloud, and Timescale). We recently rolled out HyperDX backed by ClickHouse across all clusters to reduce dependency on third-party observability systems. - AZ-aware deployments - the cross-AZ network costs associated with our service (10s of TBs of data, 100s of millions of tasks/day) would be insanely expensive. All of our infra is deployed with AZ affinity, with multi-AZ replicas and cross-region backups. If you're looking to scale and don't have super complex infra needs, using Porter will get you quite far. Otherwise, IaC is the way to go.

27

6 Comments

AWS Config costs can get out of control if you run volatile workloads with short lived Fargate tasks or EC2 instances. I reduced my Config costs with a simple trick that you should know too. We create many short lived (< 1 min) Fargate tasks in our app marbot.io. When we launch a task, an ENI is created, which causes 4 changes to be recorded by Config (ENI, VPC, subnet, security group). 4 AWS Config changes cost $0.012. Running a Fargate task for 1 minute costs $0.0008 (1 vCPU with 2 GB memory). In other words, we pay more for AWS Config than for Fargate which is crazy :) By switching the Config recording frequency from continuous to daily we reduced our Config bill significantly (see chart). Learn more here: https://lnkd.in/gPNpiscr

182

29 Comments

New Blog Article: "The Standard Terraform and OpenTofu Files + Their Uses" 📂 📂 📂 This topic should be a "TF 101" sort of thing, but we see clients get it wrong a lot: 1. They shove too much into `main .tf` OR they create 50 different files in a single root module (likely meaning they've shot themselves in the foot with a Terralith) 🦶🔫 2. They create 10 `locals` blocks in a single file instead of organizing them sanely 🤢 3. They don't know how to use the lock file properly and it never gets checked in 🔓 And a dozen other small, but productivity killing anti-patterns. This article provides answers to all those things and a lot more. It may be rudimentary, but it's worthwhile knowledge and is definitely a definitive resource. Check out the blog post at the link in the comments. Read it, internalize it, implement it, and prosper ✨ Let me know your thoughts in the comments. Is there anything we missed? Anything you disagree with? #terraform #opentofu #infrastructure #infrastructureascode #iac #platform #platformengineering #platforms #cloud #devops

42

17 Comments

🚀 Just wrote about our latest release - vCluster Auto Nodes and how it delivers dynamic #Kubernetes scaling across any infrastructure! Powered by ⚙️ #Karpenter, Auto Nodes revolutionize dynamic provisioning by offering versatility your enviroments: ☁️ Public Clouds (AWS, GCP, Azure) 🏢 Private Clouds (OpenStack, MAAS, etc.) 🖥️ Bare Metal compatibility with KubeVirt or NVIDIA BCM 👉 Read more about the vCluster Platform v4.4 and #vCluster v0.28 in my post and how things work under the hood https://lnkd.in/gxZvUKy8

40