Lefteris Karageorgiou’s Post

ICYMI, we just shipped two new open-source tools for AWS Lambda Managed Instances: 🧮 LMI Pricing Calculator — compare LMI costs against standard Lambda before you commit ⏰ LMI Scheduler — schedule-driven cost optimisation for your LMI capacity providers Both are available now in our GitHub repo, alongside the Monte Carlo simulation sample from Debasis R. 👉 Check out the tools folder under (PRs welcome!) https://lnkd.in/g-T4H5Mc #AWS #Serverless #AWSLambda #CloudComputing

Just published two new deep-dive guides as part of my CKA learning series🚀 📄 Article 1 — Setting Up a Multi-Node Kubernetes Cluster on AWS Using kubeadm - https://lnkd.in/gM93FJHj 📖 The guide covers: • Provisioning 3 VMs with the right Security Group rules • Installing containerd, runc, CNI plugins, and kubeadm on all nodes • Initializing the control plane and installing Calico as the network plugin • Joining worker nodes to the cluster using kubeadm join • Verifying the cluster and a handy troubleshooting reference 📄 Article 2 — Understanding Storage in Docker and Kubernetes - https://lnkd.in/gEAUBEsW 📖 This one covers: • Docker image layers, writable container layer and how storage drivers work • Persisting data in Docker using named volumes and bind mounts • Kubernetes emptyDir, Persistent Volumes and PVCs with access modes • Storage Classes and dynamic provisioning with cloud providers These guides are designed to provide both conceptual clarity and hands-on understanding for real-world scenarios. #Kubernetes #Docker #DevOps #AWS #CKA #kubeadm #CloudComputing #ContainerOrchestration

Day 64 of #90DaysOfDevOps – Terraform State Drift Today I learned something very real-world What happens when someone changes infrastructure directly from AWS Console? That’s called State Drift I simulated it by: ✔️ Changing EC2 Name tag manually ✔️ Running terraform plan Boom — Terraform detected the difference! Then I fixed it using: terraform apply Everything was restored back to my config Key Learning: Terraform = Source of Truth Manual changes = ❌ Not allowed in production How real teams prevent this: Restrict console access Use CI/CD pipelines for all infra changes Run regular drift detection This is exactly why Infrastructure as Code matters! Git Repo: https://lnkd.in/gu9iDuiV #90DaysOfDevOps #DevOpsKaJosh #Terraform #AWS #IaC #CloudComputing #LearningInPublic

I recently finished TicketCore - a high-load event booking API designed to handle flash sales, double-spending, and efficient cache invalidation. The original plan was a full production-grade deployment on AWS EKS using Terraform. However, I quickly hit the "FinOps Wall." Attempting to run a Kubernetes control plane, Postgres, Redis, and a Prometheus observability stack on AWS Free Tier instances is a recipe for constant OOM kills. Instead of spending $150+/month on idle cloud capacity for a stress test, I took a pragmatic approach. I kept the Terraform manifests in the repo as proof of IaC, but shifted the actual benchmarking to my local M1 MacBook using Docker Compose to eliminate virtualization overhead. The architecture held up. By offloading inventory gatekeeping to Redis Lua scripts and implementing O(1) versioned-key cache invalidation, the system sustained ~2,500 RPS with a 0% error rate under simulated flash-sale conditions. I wrote a short technical breakdown about the architecture, the database bottlenecks, and this FinOps pivot on Medium: https://lnkd.in/dSwFNPgd Сode: https://lnkd.in/dGiKuBCk

6 tabs: AWS Console. Grafana. Terminal. Logs. Docs. Slack That was my morning Now I open Cursor and just write in English: debug, scale, set alerts, check metrics. I don’t leave the editor 62 infrastructure tools: One MCP We built it at Quave ONE and I’m not going back

Just extended my Terraform infrastructure project to provision a real  AWS EKS cluster with a single command. terraform apply creates: → EKS control plane in eu-west-2 → IAM roles and policy attachments → t3.micro node group → kubectl configured automatically terraform destroy removes everything cleanly in under 10 minutes. The same codebase now targets both local Kubernetes via Docker Desktop  and cloud Kubernetes on AWS EKS. Switching between them is a single  variable change. This is what infrastructure as code actually means - reproducible,  version controlled, and destroyable on demand. No clicking through  consoles, no orphaned resources, no surprise bills. Source: https://lnkd.in/dve_set5 #terraform #aws #eks #kubernetes #devops #infrastructureascode

I think that #Kubernetes is overkill for 90% of projects, but this post seems me clickbait. Yes, #K8S maintenance is a mess (I'll post soon about my experience with Karpenter upgrade), but 1) "NAT Gateway data processing: $2,100/month" or "Load balancers (more of them): $3,800/month" definitely not related to Kubernetes 2) Docker Composer isn't answer for scale and high-availability "Our monthly #AWS bill with #EKS: $31,400. ... The Migration Back to Docker Compose ... New AWS bill: $19,100/month. We saved $12,300/month. $147,600/year."

I've been experimenting with zio-blocks to build a DynamoDB codec. The premise is simple: encode Scala case classes directly to AWS SDK AttributeValue with zero intermediate representation. No wrapper ADTs, no extra map conversions, no allocation overhead. zio-blocks' register-based architecture makes this possible — field access is a direct array index, giving us O(N) single-pass encode/decode instead of the O(N²) map-merging approach common in existing libraries. Still early days, but the core is working and wire-compatible with existing DynamoDB data. If you're in the ZIO + DynamoDB space, I'd love to hear what pain points you've hit. https://lnkd.in/e5SCjuwq #Scala #ZIO #DynamoDB #FunctionalProgramming #OpenSource

Maciej Tarsa recently built a data pipeline to ingest data from various APIs into Amazon Web Services (AWS), making the data more accessible and able to query by multiple teams in the organisation. Apache Iceberg was used to provide a structured layer over the raw S3 data however, to effectively deploy this in a native AWS environment required several architectural decisions and tradeoffs. Find out how Maciej navigated these, and the solution he ultimately implemented in our latest blog: https://lnkd.in/gDbM3MhN #AWS #ApacheIceberg #DataAnalytics

Data Engineering Journey — Iteration 4 (Handling Concurrency + Duplicates) After stabilizing my ingestion pipeline with Step Functions orchestration, I hit another real-world problem: - Concurrency spikes + duplicate ingestion. Scenario: Multiple files land simultaneously in S3 → EventBridge emits bursts of events → Parallel executions start → Retries + at-least-once delivery introduce duplicates Even with orchestration in place, the system wasn’t truly production-ready yet. So I redesigned the architecture again. New Architecture: S3 → EventBridge → SQS (buffer) → Lambda → Step Functions → Glue → Silver S3 Why this change mattered: - SQS absorbed traffic spikes (backpressure control) - Lambda controlled execution rate - Step Functions ensured orchestration reliability - Glue handled transformation cleanly But I also realized something important: - Event-driven systems MUST be built assuming duplicates WILL happen. So I added a defensive layer inside the ETL: • After transformation + cleansing • Before writing to Silver I implemented row-level deduplication: → Enforced uniqueness using review_id → Logged before/after counts for validation → Verified using Athena post-ingestion Result: - No duplicate rows in Silver - Retry-safe ingestion - Replay-safe architecture - Production-grade reliability Big takeaway from this iteration: - Scalability is not just about handling load - it’s about controlling flow + ensuring correctness under failure. This version felt like a real shift from “working pipeline” → “production-ready system.” You can take a look at my notes and my code if you want from my github repo  https://lnkd.in/dKsjJ-UZ #DataEngineering #AWS #EventDrivenArchitecture #SQS #StepFunctions #BigData #CloudEngineering