Ping Zou

Google Cloud Database Digest 30 May 2025 - Google Cloud has introduced the AlloyDB AI Query Engine (preview), enabling generative and multimodal AI capabilities directly within the database engine for lower latency and simplified application architecture. - Datastream now supports MongoDB as a source (preview), allowing real-time operational data replication from MongoDB into BigQuery for analytics without custom pipelines. - Cloud SQL instances can now be created with both Private Services Access and Private Service Connect enabled (GA), offering a unified and secure networking model for enterprise needs. - AlloyDB and AlloyDB Omni have received updates, including a new Kubernetes operator (v1.4.1 GA) with improved replication features and support for longer cluster names, plus the addition of the `pg_ivm` extension for incremental view maintenance. - Spanner now supports more efficient incremental backup copying, reducing redundant storage and network costs when copying backups between instances. https://lnkd.in/gUFQA34Q

18

1 Comment

AWS Athena ➡️ Trino: How we've scaled data infrastructure We're handling terabytes of information. And when infrastructure costs began to climb, our engineering team evaluated how to scale more sustainably while maintaining high performance. ⚙️ See Luan Carlos Araldi, Senior Data Engineer and Tadeh Alexani, Senior DevOps Engineer at adjoe, chat about the transition from using AWS Athena to managing an in-house Trino cluster. By running Trino on our existing Kubernetes clusters, the team gained better control over resource allocation and query distribution. 🛠️ The implementation includes a Trino gateway to manage multiple clusters, allowing the team to: ↳ Route queries based on specific use cases ↳ Balance loads across the infrastructure ↳ Support ad-hoc analysis alongside automated pipelines ↳ Reduce overall cloud expenditure Check out the full technical breakdown on our engineer blog: https://lnkd.in/dFyQqQNd Watch the complete bit summit presentation for a deeper look into the architecture: https://lnkd.in/dSgAvaMf #DataEngineering #DevOps #Trino #CloudInfrastructure #Kubernetes #DataScaling #BigData #adjoe

70

2 Comments

We’re excited to announce a new AI-native compute service co-engineered with Microsoft delivered as a fully managed, first-party offering on Azure. Now in private preview. This marks the start of a deeper partnership between Anyscale and Microsoft, focused on making AI-native computing faster, simpler, and more accessible for every Azure enterprise customer. As part of this partnership, we’re co-developing a deeply integrated Anyscale experience on Azure that lets you: - Deploy and manage Anyscale directly from the Azure Portal - Pay through Azure billing using your Microsoft commitments - Authenticate seamlessly with Microsoft Entra ID Under the hood, Anyscale clusters run in the customer’s Azure Kubernetes Service (AKS), right alongside their existing Azure-native storage, networking, and other workloads – to help ensure security and compliance are built into the compute foundation of enterprise AI. Learn more on how to get access and get started in our announcement blog: https://lnkd.in/gPD9K5SH

185

5 Comments

Introducing pgraft: Raft Consensus Protocol for PostgreSQL Achieve true high availability with automatic leader election, zero split-brain scenarios, and crash-safe replication. Built on the same etcd-io/raft engine that powers Kubernetes. Production-ready for PostgreSQL 16, 17, and 18. #PostgreSQL #HighAvailability #RaftConsensus #DatabaseClustering #DistributedSystems #pgElephant #DevOps #SRE #DatabaseEngineering #OpenSource #CloudNative #Kubernetes https://lnkd.in/gcMDmRfi

1

💡 What does it take to run PostgreSQL at hyperscale? OpenAI’s answer: smart schema governance, read replica optimization, and deep collaboration with Microsoft’s PostgreSQL team. At POSETTE 2025, Bohan Zhang from OpenAI shared how the team scaled Azure Database for PostgreSQL–Flexible Server to support ChatGPT’s globally distributed, read-heavy workloads. Their success with Microsoft Azure is a powerful validation of our vision for mission-critical workloads.   Explore the lessons and strategies that make PostgreSQL a backbone for AI. 🔗 Read the blog: https://lnkd.in/etEBw8yH #PostgreSQL #Azure #OpenAI #Microsoft

37

2 Comments

An expanded benchmark covering 15 #MCP server implementations across Rust, Java (Spring MVC, WebFlux, Virtual Threads), #Quarkus, Micronaut (JVM and GraalVM native images), Go, Bun, Node.js, and Python. https://lnkd.in/d4ndxGUP

16

2 Comments

From fine-grained role management to cross-region replication, from Kafka-like subscriptions to MQTT streams, TDengine makes sure your time-series data reaches the right people and systems, in real time. Read more: https://lnkd.in/e4W4hbAQ

6

🚀In the #DSS2025, Peter (Zhenqiu) Huang from Uber and Shiyan Xu from Onehouse dive deep into how #Uber is modernizing its Cloudlake with #ApacheFlink Streaming Ingestion and #ApacheHudi—achieving fresher data, simpler pipelines, and better cost efficiency. 🌟 Highlights include: ✅ Why real-time ML requires <15-minute freshness ✅ Flink vs. batch ingestion: cost, efficiency, and architecture ✅ Key tradeoffs in commit size, sorting, and file handling ✅ Autoscaling + DR for large-scale streaming pipelines ✅ How Hudi’s MOR tables unlock high-throughput streaming ▶️ Watch the recording now!
https://hubs.ly/Q03XZq6Y0 #DataStreaming #RealTimeData #DSS #datastreamingsummit

8

AWS has announced vector search for Amazon ElastiCache, enabling high-performance similarity search and retrieval for AI and ML applications via Chaitanya Nuthalapati, Chenhe L., Christoph Körner, and Ohad Shacham on Amazon Web Services (AWS) https://lnkd.in/d5tqjjZm #aws #AWSCloud #ElastiCache #VectorSearch #AI #MachineLearning #Database #CloudComputing

3

LiveRamp transformed its architecture for real-time data with SingleStore. The Chief Architect, Kannan D.R., discussed migrating to a scalable, cloud-native platform—enabling secure processing and cost savings. 🎥 Watch the recording on demand: https://ow.ly/yZni50XgXC2

96

1 Comment

Master Kubernetes-native AI infrastructure. Scale GPU workloads reliably using KAITO, liteLLM, and Flex Nodes for event-driven inference and Day 2 operations. By Sachi Desai, thanks to Cloud Native Computing Foundation (CNCF)

1

🔥Starburst Galaxy Kafka Streaming ingestion is on fire! 📊 New third-party Iceberg Kafka streaming ingestion benchmark results are out, and they provide strong independent validation that Starburst Galaxy data ingestion is highly performant and cost-effective compared to competitors. Here’s the breakdown: 1️⃣ In the benchmark, Concurrency Labs’ Ernesto Marquez compared Starburst Galaxy Data Ingestion with Amazon Web Services (AWS) AI Data Firehose and Confluent Tableflow for streaming ingestion from Kafka into Apache Iceberg using Amazon S3. 2️⃣ The tests used a standard TPC-DS dataset and evaluated ingestion from both Amazon MSK and Confluent Cloud under identical conditions. 3️⃣ The results show Starburst Galaxy resulted in an approximately 7x higher record ingestion rate compared to AWS Data Firehose or Confluent Tableflow. 4️⃣ On cost, Concurrency Labs estimates Starburst ingestion to be 87% lower than AWS Data Firehose and 81% lower than Confluent Tableflow at the tested throughput, while also producing significantly smaller Iceberg tables due to higher compression. 👉 Bottom line: If you are building Kafka to Iceberg pipelines and care about performance, cost, and open lakehouse architectures, this third-party analysis is worth reading and makes the case that Starburst Galaxy is the best possible way to implement Kafka streaming into Iceberg. Read the full benchmark results: https://okt.to/XhTSf7 #StarburstGalaxy #ApacheIceberg #Kafka #StreamingIngestion #Lakehouse #DataEngineering #Benchmarking #CostOptimization

79

5 Comments

𝗧𝗵𝗲 𝗺𝗼𝗺𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗰𝗵𝗮𝗻𝗴𝗲𝗱 𝗶𝘁 𝗮𝗹𝗹. Eric Johnson shares how one look at AWS Lambda made everything about compute click. 𝗪𝗮𝘁𝗰𝗵 𝘁𝗵𝗲 𝗿𝗲𝗲𝗹 𝘁𝗼 𝗵𝗲𝗮𝗿 𝘄𝗵𝗮𝘁 𝘀𝗽𝗮𝗿𝗸𝗲𝗱 𝘁𝗵𝗲 𝘀𝗵𝗶𝗳𝘁. 𝗙𝘂𝗹𝗹 𝘀𝘁𝗼𝗿𝘆 → https://antt.me/BXdbJK4l 𝗦𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲 → https://lnkd.in/gZ-_4tVB #AWSLambda #ModernDevelopment #TechInnovation #AntStack #Developers #ReelTalk #AntStackTV

3

Learn how to harness CPUs for practical, cost-effective MLOps workflows — Ehssan Khan unpacks the steps you should take when working with Google Cloud.

2

Yue Yin, Software Engineer at ByteDance, explains how the Gödel scheduler differs from the Kubernetes scheduler through its architecture. She details how Gödel introduces scheduling units for batch processing and implements a two-layer scheduling framework: job-level for filtering nodes, and pod-level for detailed placement. The system uses a distributed architecture with three components — dispatcher, scheduler, and binder — enabling parallel processing through multiple scheduler shards. Watch the full episode: https://ku.bz/lMpNng_33

1

1 Comment

It’s been the headline of the week: 𝐈𝐁𝐌 𝐢𝐬 𝐚𝐜𝐪𝐮𝐢𝐫𝐢𝐧𝐠 𝐂𝐨𝐧𝐟𝐥𝐮𝐞𝐧𝐭. A big question on everyone’s mind is how this will reshape the streaming landscape. In Yingjun Wu’s words, this acquisition does not mark the end of anything! It only accelerates a shift that was already in motion. The whole data infra stack is moving upward, away from raw plumbing and toward products that deliver visible value. This shift shows up in three places: ▪️ How analytics vendors treat ingestion ▪️ How Apache Iceberg changes the role of Kafka ▪️ How Kafka itself has to evolve. For details, read: https://lnkd.in/gXUMRyPA #risingwave #streamprocessing #datainfra

5

We pushed PostgreSQL to its limit: 1 Petabyte of data. Michael Zhilin from Postgres Professional details the findings, bottlenecks, and architectural changes required to handle extreme volume in a distributed PostgreSQL environment. See what happens at the very edge of scale. #ExtremeScaling #DistributedDatabase #PostgresPro

23