Tim Spann

Databricks Cluster Sizing Made Simple | QuantiByteX Getting cluster sizing right is one of the easiest ways to improve performance and reduce costs in Databricks. Yet it’s also one of the areas teams struggle with the most too many variables, too many defaults that don’t really fit real workloads. To make things easier, we put together this Cluster Sizing Cheat Sheet covering the essentials: ✔ Partitioning strategy ✔ Shuffle tuning ✔ Executor/core distribution ✔ Memory planning ✔ Photon usage ✔ Skew handling These are practical, battle-tested guidelines our engineering teams use when designing or reviewing data workloads. Nothing theoretical just what actually works in production. At QuantiByteX, we focus on building efficient, scalable data systems, and smart cluster sizing is a big part of that. If your team is trying to optimize pipelines, cut cloud costs, or tune slow jobs, this cheat sheet is a solid place to start. #QuantiByteX #Data #Science #Databricks #Cluster #Cloud #Bigdata #Optimization

4

💰 Up to 75% lower Snowflake costs. We benchmarked traditional ELT pipelines against a streaming-first approach with DeltaStream, and the results were clear: massive savings, faster data, and leaner warehouses. The key? Shifting left, processing and optimizing data earlier in the pipeline so less work (and less cost) hits Snowflake. In our latest blog, we break down 7 proven strategies to cut Snowflake spend without sacrificing performance, from streaming CDC to warehouse-less ingest via Snowpipe Streaming. 🔗 Read the full blog here https://lnkd.in/gRmrmt6h #Snowflake #Streaming #ETL #DataEngineering #CostOptimization

8

Love Airflow but tired of the detective work, missed SLAs, and surprise failures? AAI by Broadcom connects the dots. It maps your DAGs to real business processes, tracks true SLA deadlines (no more finger math), and predicts failures before they happen — so you can stay ahead and look sharp doing it. Check out the video and dive into the AAI learning path to get started. https://lnkd.in/gjFNMqgY

38

Databricks Lakebase: The Architecture Powering Agentic AI Our VP of Data + AI, Josue “Josh” Bogran, sat down with Reynold Xin (Cofounder, Databricks) for a fast, technical breakdown of Lakebase , and why it matters for the future of AI-driven applications. In this quick session, they discuss: ✅ Why OLTP databases haven’t evolved in decades ✅ How Lakebase enables cloud-native, agent-native Postgres ✅ Separation of storage & compute using open data lake formats ✅ Millisecond autoscaling and zero-compute architecture ✅ How Lakebase differs from Aurora, and why it matters ✅ A live demo of branching, snapshots, and instant provisioning 👉 Watch the full conversation: https://lnkd.in/gn8ZwJdU #Databricks #Lakebase #AgenticAI #DataAndAI #Postgres #Lakehouse #zebDataAI

9

I’ve been testing both approaches, and the difference is pretty clear. Some tools still stop at “here’s how to do it” while Cortex Code actually builds and delivers the outcome. That shift from guidance to execution is what really stands out.

2

If you are still on Hadoop and are looking to move off, here are real world lessons learned that can save you time, money and your reputation.

14

ICYMI - tons of great content on the IDUG blog! Find these recent articles at idug.org/news: - Intelligent Anomaly Detection and Resolution in IBM Db2 Genius Hub by Jobin J - Datalake Tables in Db2: Open, Flexible Access to Your Data Wherever It Lives by Dominic So - The Bit that Broke Db2 by Marcus Davage - Db2 z/OS – vNext: Synonym remediation by Scott Walker

15

Performance benchmarks don’t lie. We ran 25 real-world queries across Firebolt, Snowflake, Redshift, and BigQuery. Firebolt’s smallest engine outperformed every config tested - even Snowflake XL. Same data, same queries, same region. ▶️Watch: https://lnkd.in/dn6VZ7kg 📝Read: https://lnkd.in/dYYV6e_Y #dataengineering #benchmarks #cloudatawarehouse

37

Feature engineering is often the most critical (and complex) part of ML. This guide provides a clear introduction to Snowflake Feature Store and how to structure entities, feature views, and datasets for model training and inference.  A helpful resource for data scientists and ML engineers looking to standardize and scale feature workflows.

8

Orchestration is not optional. We’re joined by Egor Tarasenko at Ponder Labs to talk about why every ingestion path, whether it touches dbt Cloud, BigQuery or custom pipelines, ultimately routes through Airflow. We explore how centralizing triggers inside a single DAG ecosystem creates visibility across dependency datasets and upstream failures. It’s how teams maintain structure, consistency and control as multi-tenant data platforms scale. Click the link in the comments for the episode. #AI #Automation #Airflow

13

5 Comments

Terraform provisions Snowflake. That does not solve the harder problem of governing schema and control plane change over time. As Snowflake takes on a bigger role in analytics, data products, and AI, teams need more than provisioning. Our latest post breaks down where Terraform fits, where Liquibase Secure fits, and why provisioning is not enough. Read it here: https://hubs.li/Q047Ckpr0 #Snowflake #DataEngineering #DataGovernance #DevOps #Terraform

10

Bring reliability and trust to your Gen AI apps! AI Observability in Snowflake Cortex AI is now generally available, empowering developers to evaluate and optimize. 👩‍💻 Observability is deeply integrated across Cortex features like Search, Analyst, and Document AI. Discover how Snowflake builds foundational capabilities to manage, improve, and trust your AI apps end-to-end. ❄️ Learn more: 👉 https://bit.ly/3UcrpAv

180

3 Comments

Great news, Snowflake Cortex Agents API is now GA! Developers can now build trusted, production-ready data agents that orchestrate insights across structured and unstructured data using Cortex Search and Cortex Analyst. Build context-rich agents today: https://lnkd.in/d7GNsEwj

3

Get started with the Snowflake managed MCP Server! Learn how to build AI agents that securely access Snowflake data through the Model Context Protocol - a standards-based approach that eliminates custom integration complexity. Key technical advantages: • Governed by design: Leverage existing RBAC and data masking policies • Reduced integration overhead: One connection enables any MCP-compatible agent • Extensible architecture: Out-of-the-box access to Cortex Analyst and Cortex Search This quickstart walks you through configuring Cursor to interact with Snowflake MCP Server, enabling your agents to reason over both structured and unstructured data with enterprise-grade governance. The result? Faster deployment of generative AI applications with richer data insights and simplified application architecture. 🔗 Get started with the quickstart from Dash DesAI: https://lnkd.in/gz64vMeZ

3,467

23 Comments

There’s a new way to move dbt Labs projects to Databricks—one that turns months-long projects into weeks-long engagements. dbt's portability promised platform flexibility, but migrations still required massive SI costs or endless internal engineering hours. We've eliminated that barrier. Connect your systems, hit start, and get production-ready validated code in weeks. And since Datafold is a Databricks Delivery Partner, customers can engage through existing agreements—no separate contracts or security reviews. How’s that for fast? https://lnkd.in/gdczx-36

40

1 Comment

Switching between tools is still a big part of data workflows. This integration brings Databricks into Cortex Code, making it possible to explore, build, and debug without leaving the same environment. More from Dash DesAI: https://lnkd.in/g9zrHF_P

3