ScyllaDB’s Post

Obligatory AWS post below: Seriously though, yesterday’s AWS outage was a good reminder that resilience isn’t optional. Even the most reliable cloud can fail. And when it does, it tests every assumption in your architecture. Multi-region, active-active, data partitioning, and built-in failover aren’t just "nice to have" anymore… they’re survival tools. If one region going dark can stop your system, you're not looking at a cloud problem, you got a design problem. How do you build resilience into your real-time systems? #Resilience #RealTimeData #CloudArchitecture #Hazelcast #AWS

🏅Day 1 of Cloud & DevOps Blog Series – The Cloud Bill That Spiked Overnight * Everything was fine on Friday. * By Monday morning, the cloud bill had spiked 5x higher. * No new projects. No big deployments. Just… a shocking invoice. ## The Hidden Culprit: Orphaned Resources ## In my case, the spike came from: - Compute instances left running after testing. - Idle BigQuery slots reserved but never used. - Load balancers without traffic but still billing. - Storage buckets with forgotten debug logs. The scariest part? None of these triggered alerts, because technically, everything was “healthy.” How we fixed it: We built cost governance guardrails: + Automated scripts to detect and clean up unused resources. + Budget alerts tied to projects & teams. + Labels/tags for cost attribution and accountability. + Dashboards in BigQuery + Looker for near real-time spend visibility. Lesson learned: Cloud doesn’t punish you for mistakes with errors, it punishes you with invoices. $ Always monitor cost as a first-class metric, not an afterthought. Over to you: Have you ever been surprised by an unexpected cloud bill? What was the strangest hidden cost you uncovered? #Cloud #DevOps #AWS #GCP #Azure #FinOps #CloudCostOptimization #Kubernetes #Terraform #SRE #CloudComputing #Infra #Platform #CICD

We benchmarked a 2TB Dragonfly Swarm cluster. ✅ 10 million+ requests/sec (RPS) easily ✅ Nearly 20 million RPS with pipelining ✅ Far more cost-effective than traditional solutions It’s time to change how we think about scaling in-memory data infrastructure. Read the deep dive: https://hubs.la/Q03LslNQ0 #Cloud #Database #Performance #Scalability

GKE Regional vs. Zonal Clusters and Node Pools I wanted to summarize the results of my recent poll and clarify the correct answer. The correct answer was 3, and it turns out not many people got it right. The key lies in understanding the distinction between regional and zonal configurations for both GKE clusters and their node pools. First, it's crucial to know that a cluster and its node pools must share the same location field; otherwise, you'll encounter an error. For the cluster itself, this location field determines where the control plane nodes are placed. If you specify a region, GCP will deploy the control plane nodes across three zones within that region for high availability. If you specify a single zone, they will all be placed in that one zone. Now, things get more interesting with node pools. They have an additional field called node_locations, which overrides the default placement and specifies the exact zones for the worker nodes. If you don't define node_locations, the worker nodes will be deployed in the same zones as the control plane. Here's the final piece of the puzzle that was critical for the poll question: the node_count parameter (as well as min_node_count/max_node_count in an autoscaling configuration) defines the number of nodes per zone, not the total count for the entire node pool. Also, remember the importance of the remove_default_node_pool setting in the google_container_cluster resource, which allows you to delete the default node pool upon creation. #terraform #gcp #devops #cloud #kubernetes #gke

Something very interesting happened earlier this week and as an AWS and Cloud enthusiast it encouraged me to turn to Linkedin to share my two cents as well. AWS’s Northern Virginia region (US-East-1) - a key region serving majority of the Global Cloud market) went down after a DNS and DynamoDB failure which cascaded across multiple dependent services like EC2, S3, Lambda Functions and more taking major platforms/websites offline resulting in hundreds of millions if not billions of dollars in lost revenue. What stood out to me was not just the outage itself, but how it happened despite AWS’s layered redundancy. Even with backups of backups of backups, the failure occurred at a core dependency layer (DNS + service discovery) that every other service relied on proving that resilience is not just about replication, but dependency isolation. As someone working on cloud migrations and systems designs/architecture, this reinforced a few truths: - Redundancy is not Resilience. We must design for isolation, not duplication. - Multi-region strategies must be actively tested, not just architected. - Dependency maps should be as critical as the data backup plans. - When designing infrastructure architecture - Use the Design for failure assuming it will eventually happen. In today's world it is not if it will happen but when. Cloud reliability is not about avoiding outages. It is about ensuring your business survives one with appropriate plans in place. #AWS #CloudArchitecture #Resilience #AI #DigitalTransformation #SystemsDesign

So, after this week’s AWS outage: > Everyone: We should go multi-cloud! > Me: Cool. Can you also replicate your data, infra, IAM, telemetry, CI/CD, and SLAs across providers? > Them: “...” 😅 Multi-cloud sounds smart on paper — until you try to make it real. > Redundancy ≠ resilience. Sometimes the right answer is better architecture, not more clouds. #Cloud #Azure #AWS #DevOps #Resilience #Architecture

Every time AWS sneezes, half the internet catches a cold. 😅 Today’s AWS us-east-1 outage is another reminder that “the cloud” isn’t magic — it’s just someone else’s data center. We love to say our systems are resilient, but most are just optimistic. Let’s be honest — we often design for when things work, not when they break. True resilience isn’t built by trusting the cloud. It’s built by expecting failure and surviving it. Because at the end of the day — hope is not a failover strategy. ⚡ #AWS #CloudComputing #USEast1 #DevOps #Resilience #Engineering #Architecture #SRE #SiteReliability #CloudArchitecture #SystemDesign #Outage #TechHumor

Is the cloud expensive at scale or are we asking the wrong questions? 🤔 “No architecture can save a bad business model" Evandro Pires explains why if costs grow while revenue doesn’t, the problem isn’t JUST technical: https://lnkd.in/gFzPkZB8

“We scaled to the cloud — and our AWS bill became the real outage.” We thought we were scaling “smart.” Auto-scaling groups. Microservices. Kubernetes. Everything cloud-native. Until the invoice arrived. The problem Our monthly AWS cost jumped from $3,200 → $27,000 — in 3 weeks. Traffic hadn’t grown. Just our complexity had. Here’s what went wrong: • Every microservice had its own ECS cluster. • Logs were being shipped to CloudWatch at 20 GB/hour. • Each service had provisioned RDS, not serverless. • We used Lambda + API Gateway for tiny jobs — but each call was billed. We’d built an elegant system… That bled money silently. The fix We stopped “scaling everything” and started measuring impact. • Merged low-traffic services into one shared node • Added S3 + Athena for cheap log analytics • Switched to Aurora Serverless v2 • Cached heavy endpoints with CloudFront + Redis • Set up FinOps monitoring (CloudZero + Budgets) Cost dropped by 70%. Performance? Actually got better. The takeaway Scaling isn’t about adding more servers — it’s about adding more sense. Cloud makes it easy to scale, but even easier to waste. If you can’t explain your cloud bill, you don’t really understand your architecture. Have you ever faced a “cost outage” before? How did you bring it under control? #SystemDesign #CloudArchitecture #FinOps #AWS #EngineeringLeadership #DevOps #Microservices #Scalability #Architecture #CostOptimization

The Great AWS Debate: Does Lambda Really Run Inside a VPC? 🤔 It’s one of those classic AWS trick questions.... “Does Lambda run inside your VPC or not?” The confident ones say, “Yes, of course, you can attach it to your VPC.” The seasoned ones smirk and go, “its complicated...!"😏 Here’s the truth behind the confusion 👇 👉 Lambda never truly runs inside your VPC. 👉 It lives in an AWS-managed VPC that hosts all Lambda infrastructure. 👉When you attach it to your VPC, AWS quietly creates Elastic Network Interfaces (ENIs) inside your subnets and that’s how it connects to your private resources (RDS, Redis, etc.). You control the ENIs’ security groups and routes, not the Lambda’s actual runtime network. Once attached, the function loses public internet access.. So you’ll need NAT Gateways or VPC Endpoints for outbound calls. Cold-starts get a bit heavier too, since AWS must spin up ENIs and warm network paths. So technically, Lambda interacts with your VPC but executes elsewhere inside AWS’s own invisible one. Why the special treatment? 🤔 Because Lambda’s design goal is isolation and scale. Each invocation runs in a micro-server, optimized for security and speed across regions. If AWS actually dropped those runtimes inside customer VPCs, they’d lose the flexibility to scale millions of concurrent invocations instantly. So yeah... Lambda and VPC have a complicated relationship. They’re connected, but not committed. #AWSLambda #AWSVPC #Serverless #CloudComputing #DevOps #TechHumor #CloudMemes #ItsComplicated