Randy Pitcher 🌞

Databricks set a new direction at the 2025 Data + AI Summit. The new mission is clear: build and deploy full-scale data and AI products on a unified, AI-native platform. Here’s what’s new: • Lakebase – Real-time workloads, fully integrated • Agent Bricks – Build, deploy, and manage AI agents • Lakeflow + Declarative Pipelines – From pipeline code to pipeline intent • Lakebridge – AI-powered onboarding, accelerated • Unity Catalog Metrics – Governed business metrics, centralized • AI/BI – Business intelligence with an AI-native interface • Databricks Apps – Serverless, data-native applications • Databricks One – Unified view of metrics, apps, and agents • Free Edition – No-cost access for small teams and prototyping We broke down what each launch means, and how they’re changing the way teams execute AI at scale. Get the details: https://hubs.li/Q03sMykq0 #DataAISummit #Databricks #Indicium #AIExecution #IndiMesh #ModernDataStack

44

We’re excited to announce full Apache Iceberg™ support in Databricks, unlocking the full Iceberg and Delta Lake ecosystems with Unity Catalog! This is a major advancement toward a single, unified open table format. Now you can: - Use Databricks and external engines to read and write Iceberg tables managed by Unity Catalog - Access and govern Iceberg tables managed by external catalogs Get the details here: https://lnkd.in/gMQ2Gd9s Tune-in to #DataAISummit to hear Matei Zaharia share more.

1,005

13 Comments

Quartr API datasets are now available via Snowflake. Access structured earnings data – live audio, real-time and historical transcripts, filings, slide decks, and event summaries. Pull Quartr data directly into your existing Snowflake workflows and augment your datasets. No new pipelines, no separate API integration. Build with data that drives decisions.

75

Agentic Data Quality Monitoring is now in public preview on Databricks, powered by Unity Catalog: https://lnkd.in/gBYMr8hi. This next generation of data quality monitoring looks at how your data is used and produced to automatically flag and prioritize the most important issues without needing lots of manual rules and thresholds. Just one example of how AI can transform all aspects of data engineering.

2,813

76 Comments

Demo: how to surface and use Unity Catalog Metric Views in Databricks AI/BI — to enable self-serve access to trusted business metrics. See how to set up central governance with row-level security and column masking, and more.

403

8 Comments

Databricks Performance Optimization: Speed That Saves Money In Databricks, performance isn’t just about faster queries — it’s about using compute wisely. When pipelines aren’t optimized, costs rise quietly in the background. So what actually makes a difference? 🔹 Smarter data layout Well-organized data means systems scan less and respond faster. 🔹 Balanced data distribution Too many splits slow things down. Too few limit growth. Getting the balance right improves both speed and scalability. 🔹 Reuse instead of reprocessing Data that’s accessed often should be reused, not recalculated every time — saving time and cost. 🔹 Right-sized compute Bigger infrastructure doesn’t always mean better performance. The right size delivers the same results at a lower cost. 📊 The business impact Faster data pipelines Lower cloud spend More stable performance as data grows Well-optimized platforms don’t just run better — they pay for themselves. 🚀 Fast data isn’t expensive data — when it’s engineered the right way. #Databricks #DataEngineering #CostOptimization #ScalableData #Lakehouse #CloudEfficiency

21

It would've taken 25 years for this Snowflake customer to get 100% business data quality coverage. With Anomalo, Discover now configures unsupervised data quality monitoring within 10-15 minutes for each dataset. Learn how Discover’s data team flipped the script using Anomalo on Snowflake to scale trust across petabyte-scale data: https://lnkd.in/gPjxDCFH

11

We've launched the dbt_semantic_view package, seamlessly integrating the Snowflake semantic view with dbt's disciplined workflow. This means you can now: - Centralize Metrics: Define key metric logic (FACTS, DIMENSIONS, METRICS) natively in Snowflake's semantic view for a single source of truth. - Enable CI/CD: Apply dbt's strengths (declarative YAML, testing, and version control) directly to your semantic layer definitions. - Ensure Governance: Automatically manage privileges and validate relationships to ensure every consumer, from BI tools to AI agents, uses trusted data. - Unify your governance framework and ensure consistent, high-performance metrics across your enterprise. Learn more and get started: https://lnkd.in/eZqiwT-g

492

2 Comments

🚨 Snowflake's Cortex 𝘈𝘐_𝘊𝘖𝘔𝘗𝘓𝘌𝘛𝘌 function now supports strongly-typed outputs! We had structured outputs but the numbers still come back as strings, booleans as text, and so on. This meant adding 𝘛𝘙𝘠_𝘊𝘈𝘚𝘛 and other validation steps downstream. But not anymore! ✅ Now, you can enforce a strict schema with 𝗧𝘆𝗽𝗲 𝗟𝗶𝘁𝗲𝗿𝗮𝗹𝘀 directly in the 𝘰𝘶𝘵𝘱𝘶𝘵_𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦. This isn't just a hint to the model; 𝘈𝘐_𝘊𝘖𝘔𝘗𝘓𝘌𝘛𝘌 now verifies each generated token against your definition to guarantee the response conforms to the specified types. 𝗧𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁 𝗶𝘀 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰, 𝗿𝗲𝗹𝗶𝗮𝗯𝗹𝗲, 𝗮𝗻𝗱 𝗰𝗹𝗲𝗮𝗻 𝗱𝗮𝘁𝗮, every single time. You can now define your desired output using familiar SQL types. Just start your type literal with the 𝘛𝘠𝘗𝘌 keyword and use a SQL 𝘖𝘉𝘑𝘌𝘊𝘛 as the top-level structure. 🎯 This simple change has a massive impact: 1. 𝗥𝗲𝗱𝘂𝗰𝗲𝘀 𝗣𝗼𝘀𝘁-𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Your data lands correctly typed from the start. No more cleanup. 2. 𝗜𝗻𝗰𝗿𝗲𝗮𝘀𝗲𝘀 𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆: Eliminates an entire class of data quality errors, making your pipelines more robust. 3. 𝗦𝗲𝗮𝗺𝗹𝗲𝘀𝘀 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻: Perfect for systems that require deterministic, schema-compliant responses. This feature is supported by every model in 𝘈𝘐_𝘊𝘖𝘔𝘗𝘓𝘌𝘛𝘌. This is a huge leap forward for building 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗴𝗿𝗮𝗱𝗲, 𝗔𝗜-𝗻𝗮𝘁𝗶𝘃𝗲 𝗱𝗮𝘁𝗮 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀. You can now build with more confidence and less defensive code. ⁉️ What other quality-of-life improvements are you using today? Pavan Mona Vivek Jessie Paritosh Boxin Nicole Jiwoong Arun Siddharth James Josh #Snowflake #DataEngineering #AI #LLM #SQL #SnowflakeCortex #Developer #DataQuality

655

20 Comments

New tutorial drop 🫳 Combine LlamaIndex's LlamaParse with Snowflake Cortex for a powerful, enterprise-grade RAG pipeline: 📄 Parse complex PDFs with state-of-the-art table extraction and multi-format support using LlamaParse 🔍 Create hybrid search with Snowflake Cortex Search for fast, accurate retrieval 🤖 Build an end-to-end RAG pipeline that can answer questions from your parsed documents ⚡ Handle everything from document ingestion to text splitting to generation in one unified workflow Check out the solution: https://lnkd.in/e2sGHyTt Or go straight to the demo video to follow-along: https://lnkd.in/ePRf8QPw

46

🚀 Databricks DQX — new release v0.13.0 available 🎉 DQX has also reached 3.5 million monthly downloads on PyPI. This release delivers major improvements, making it easier than ever to manage and monitor data quality checks. 🌟 Highlights ✅ New DQX Data Quality Dashboard Added significant enhancements to the DQX dashboard providing a centralized view of data quality metrics across all tables: https://lnkd.in/dBiwa_ig ✅ DQX App Introduced the foundation of the DQX application, including AI-assisted rule generation from natural-language input and Configuration management. This is the first version of the UI, laying the foundation for upcoming enhancements such as one-click deployment, graphical interface for building rules, saving/laoding rules from storage and execution of checks. ✅ New row-level checks: is_null, is_empty, is_null_or_empty Introducing new checks which enable verification of column values as null, empty strings, or both, complementing existing checks: https://lnkd.in/dsgTNVtX ✅ Summary metrics support on all clsuter types Summary metrics can now be used in all cluster types including Servelress and standard clusters with DBR 17.3 LTS and above. Previously limited to dedicated clusters only: https://lnkd.in/dWaK6mav ✅ DQX Best Practices New guidance for running DQX in production: https://lnkd.in/dS2SR3xz ✅ Tolerance support for equality and non-equality checks Added absolute and relative tolerance for all equality and non-equality check such as is_not_equal_to, is_equal_to, is_aggr_not_equal, is_aggr_equal 🔧 Additional Enhancements * Support for multi-line SQL expressions in sql_expression checks * Support for decimal for limits in quality rules and rules generator * Enhanced PII detection resolution for metadata-defined checks * Updated Lakebase authentication using client IDs * Added exclude columns parameter to the schema validation check to exclude columns from validation * Enhanced profiler & generator documentation * Support for filters in ODCS-generated rules 📖 Release notes: https://lnkd.in/dqYnAJaN 🛠️ GitHub: https://lnkd.in/diTgU3h2 📦 PyPI: https://lnkd.in/dRWNKnAf 📃 Documentation: https://lnkd.in/dFbWdsUT 🙌 Huge thanks to everyone who contributed to this release — your work continues to push DQX forward as a powerful, open-source data quality framework for Databricks! Marcin Wojtyczka Gregory Hansen Alex Ott 🪛 Eniwoke Cornelius Neha Milak Laurence Wells Ivan Trusov Manuel T. Pierre Monnet Vikas Katiyar Ashwin Sahay David Wanner @bpm1993 Josef Gustafsson

273

7 Comments

If you're looking for a code-first method of deploying PBIP files or semantic models, check out this article I put together on how to get started with fabric-cicd! fabric-cicd is an amazing Python library developed by my teammates that provides a code-first method to deploy Fabric items from source control to Fabric workspaces. https://lnkd.in/evUibVGT

328

15 Comments

If your team is building with LLMs, you know the value of clean, retrievable data. That's why we're teaming up with Unstructured to streamline one of the hardest parts of AI development. This integration removes friction across the AI stack—from unstructured ingestion to semantic retrieval—so your teams can focus on building. Read more from our Product Marketing Managers, Rini Vasan and Jim Allen Wallace, and Unstructured’s Head of Developer Relations, Maria Khalusova here: https://lnkd.in/e98QqXAB

43

Databricks isn’t the problem. The missing orchestration layer is. Even with millions invested, too many teams end up with: 😱 Endless engineering backlogs 😱 Fragile, inconsistent pipelines 😱 Stakeholders shut out of the process The result? Zero trust. Zero impact. In our free whitepaper, we break down the 5-step action plan to turn Databricks into real business value. Link in the comments ⬇️

26

2 Comments

Databricks Community Edition is gone, and the new Databricks Free Edition is now live 👍 What is there for you? 1. Same look and feel as the premium edition. 2. Now you can experiment and learn using almost all features (still a few exceptions). 3. A lot more learning with Data Engineering and Data Warehouse capabilities at no cost. 4. A lot more learning with GenAI capabilities without spending any money. 5. No Compute and only Serverless cluster available. Important Notes:- 1. Migrating from the community to the free edition will not migrate your notebooks. So, make sure to take your backup and export your files before migration. 2. Now fair usage policy applies, so excessive usage might shut down your resources for the day or even for the rest of the month. 🤥 Master Spark, Databricks, Performance Tuning, and GenAI with our courses at www.scholarnest.com Follow Prashant Kumar Pandey for more. Keep learning and keep growing. #Databricks #Generative AI, #ScholarNest

239

14 Comments

Snowflake Catalog Linked database. https://lnkd.in/gpYi5FZa 1. Table Created from Spark \You can add a new column from Snowflake. \You can insert data from both Snowflake and Spark. 2. Table Created from Snowflake \Don't alter the table from Spark; alter it only from Snowflake. 😮 \You can insert data from both Snowflake and Spark. \Dropping the table from Snowflake will remove the entry from the catalog. You need to manually remove the S3 directories — which is fine. 3. Reading Snowflake inserted data from Spark. \see below error Cannot support vectorized reads for column [coffee_id] optional int64 coffee_id (INTEGER(64,true)) = 1 with encoding DELTA_BINARY_PACKED. Disable vectorized reads to read this table/file 4. Reading Spark-Inserted Data from Snowflake \No issues. 5. Snowflake Does Not Honor Apache Iceberg's Partition Physical Directory Format \If you insert data from Snowflake, all data files are stored in the data directory. \Be mindful of potential AWS S3 throttling. My verdict. Don't mix Vodka and Whisky. Snowflake Ashish Mishra

16

Big news from the Astreya × Databricks universe. Data Trust & Stats Intelligence (DTSi) is now live on the Databricks Marketplace! If you work with large, messy, fast-moving datasets, this is one of those releases that actually makes your day easier. What it does: Removes the constant question of “Can we trust this data?” Replaces it with “What can we confidently decide right now?” DTSi automates data validation, surfaces statistical insights, and flags anomalies early, turning scattered, unreliable data into clear, decision-ready intelligence your teams can act on. Learn more: https://ow.ly/T8TT50XC0wt #Databricks

69

Data doesn’t live in one system, your AI tools shouldn’t either. Snowflake’s Cortex Code CLI now supports dbt and Apache Airflow, extending Snowflake’s secure, context-aware AI coding agent across modern data engineering workflows. Teams can build, debug, and optimize models and pipelines within the tools they already use, with AI that understands schemas, dependencies, and data context across systems. This is governed, enterprise-ready assistance, with administrative controls for access and policy enforcement, and model choice  including Claude Opus 4.6 and OpenAI GPT-5.2 to balance quality, latency, and cost. For data leaders, engineers, and platform teams, this means less friction across distributed environments and more confidence in production pipelines. Intelligence is moving closer to the work itself wherever the data lives. ❄️ https://lnkd.in/gDXE5wrD

152

6 Comments