Casper Lubbers’ Post

🏗️ Stop moving data. Start unifying it. The Databricks OneLake approach (via Unity Catalog) is finally killing the "extract-and-load" cycle. By decoupling metadata from the storage layer, we’re getting: • Zero-copy integration across clouds. • Universal Governance (one SQL/Python policy to rule them all). • Open Standards via Delta Sharing & Iceberg REST. The big question for the architects: Is the "Single Control Plane" actually achievable, or is "Federated Chaos" the permanent reality of 2026? 🧐 Let me know what you think👇 #DataArchitecture #Databricks #DataEngineering #UnityCatalog

👀Enterprises are at a key inflection point that has them making key decisions that move them away from “good data warehouses” towards data platforms that give them integration with a broader ecosystem, more robust security and governance capabilities, best in class ML/AI, cost controls, etc. While performance and cost are important, making decisions on these vectors alone means everything else is a commodity, and it just isn’t. 🤔Here is a great read on a company’s journey all the way from realizing their current warehouse was creating bottlenecks and headaches, through their evaluation of Databricks and Snowflake, and ultimately how they made their choice(s). 💪 I love that they had a “squad lead” for the duration of the evaluation (single-throat accountability helps clear blockers, make decisions, etc quicker), came up with their own contextual criteria, and spent a meaningful amount of time on the POC (but not so much time that the puck moves and/or you get analysis paralysis). Definitely worth the read! https://lnkd.in/g2CzaaUW

Stop Guessing. Start Triggering. 🚀 Data engineers, we’ve all been there: setting a cron schedule for 3:00 AM and "hoping" the upstream data arrived by 2:59 AM. When it doesn't, we face failed jobs, empty tables, or the dreaded manual backfill. Databricks just changed the game with Table Update Triggers. Instead of running jobs on a rigid clock, your workflows can now react instantly the moment data hits your source tables. The Old Struggle: > Idle Clusters: Spinning up compute just to check if data is there—and finding nothing. > Data Latency: Upstream finishes at 8:05 AM, but your job doesn't run until the 9:00 AM schedule. > Brittle Dependencies: Complex "sensor" logic just to coordinate across different pipelines. The New Reality: > Event-Driven: Jobs fire as soon as a Delta or Iceberg table is updated, merged, or deleted. > Multi-Table Intelligence: Only trigger once all your source tables (up to 10) have been updated for the day. > Built-in "Quiet Time": Use the Wait after last change setting to ensure a massive batch load completes before downstream processing starts. Real-World Use Cases: - Instant Bronze-to-Silver: Move data through your Medallion layers the second it lands. - Cost-Efficient ML: Trigger model retraining only when your feature table actually changes, not on a blind weekly schedule. - Smart Reporting: Refresh Materialized Views or Slack alerts immediately after your final aggregation table commits. This is a massive step toward a truly reactive data platform that saves on compute costs while slashing data latency. Time to delete those "safety" buffer hours in your schedules. 🛠️ Official page: https://lnkd.in/g6Xt33WA #Databricks #DataEngineering #ETL #Lakehouse #DataArchitecture Thank you Databricks <3

🚀 Since last week, Databricks Lakebase is now generally available. It is serverless Postgres which is integrated with the lakehouse and it is built to reduce and eliminate the delays, duplication and complexity in OLAP and OLTP systems live in separate stacks. 💡 How it will help in broader data lifecycle: 👉 Lakebase brings many key features like serverless autoscaling, instant database branching, and most importantly, point in time recovery. All of these features give teams safer, faster, and more flexible ways to build and operate transactional workloads. 👉 Governance: With Unity Catalog integrated directly into Lakebase, there is no need to manage separate security models for OLTP systems. All the access control and auditing are unified across operational and analytical data. 👉 Lineage and metadata: By reducing “shadow pipelines” and introducing governed sync relationships between Lakebase and lakehouse tables, teams get cleaner lineage, consistent metadata, and a more transparent, traceable data estate. https://lnkd.in/e_Nh4GgH https://lnkd.in/ewPv9S9a #Databricks #Lakebase #Governance #Lineage #Metadata #DataAnalytics #AI

#databricks_basics_48 🚀 How Delta Lake Enables ACID Transactions in Databricks Building reliable data pipelines requires strong transactional guarantees — and Delta Lake delivers exactly that. Here's a quick breakdown 👇 🔒 Atomicity – All or Nothing Every operation is broken into commits recorded in the Delta Log. A commit is added only when all actions succeed, ensuring no partial writes. 📏 Consistency – Always Valid State Delta Lake enforces strong schema checks, ensuring tables remain in a valid state after every transaction. 🔀 Isolation – Safe Concurrent Writes Uses optimistic concurrency control, where each write checks for conflicting updates before committing — preventing data corruption even under heavy concurrency. 💾 Durability – Changes That Stick Committed changes are written to underlying Parquet files and logged in the Delta Log, ensuring permanence even after failures. 📁 Delta Log – The ACID Backbone The hidden _delta_log stores ordered JSON commit files that act as the single source of truth for the latest table state. ⚙️ Databricks Enhancements Databricks layers ACID over cloud storage using transactional commits (_started_ & _committed_ metadata files) and provides snapshot isolation for reads + write‑serializable isolation for writes. 🔥 Bottom line: Delta Lake transforms your data lake into a trustworthy, transactional system, making analytics and ML pipelines more robust and reliable. You might remember from my earlier post that this Delta Logs itself also enables Databricks features such as time travel - the point being, delta log is backbone to databricks ecoosystem #DeltaLake #Databricks #ACIDTransactions #DataEngineering #BigData #DataLakehouse #OptimisticConcurrency #DataReliability #DataConsistency #Parquet v4c.ai#TransactionalData #DistributedSystems #DataPipelines #AnalyticsEngineering #LakehouseArchitecture #Spark #DataOps #ETL #MachineLearningPipelines #EnterpriseData

🚄 Databricks Lakebase is now Generally Available Key capabilities available today include: 💯 Serverless autoscaling and scale to zero: Compute resources dynamically adjust to match traffic spikes and shut off completely when idle to eliminate wasted costs. 💯 Instant database branching: Create isolated, zero-copy clones of production data in seconds for risk-free testing and development. 💯 Point-in-time recovery (PITR): Protect against accidental deletions or bugs with millisecond-level restoration. 💯 Unified governance: Manage access control and auditing through Unity Catalog for a single security model across your entire data estate. 💯 Sync tables: Keep your operational data and historical lakehouse context in sync without maintaining fragile pipelines. Learn more: https://lnkd.in/ec2mGpQA Susan Pierce Yanic M. Bilal Obeidat #Databricks

Common Databricks Pipeline Mistakes Just because your pipeline runs doesn't mean it's production-ready. Here are the mistakes that come back to cause the issues later. 1) Poor Delta Lake Maintenance Every time you write data, Delta Lake creates small files. Too many small files equals slow queries. The solution for this is to run OPTIMIZE weekly (or daily) on your busy tables to merge these files together. 2) Inefficient Cluster Management Stop using always-on clusters for scheduled jobs. Switch to job clusters that spin up, run your task, and shut down automatically. This can cut your computing costs in half. 3) Missing Incremental Processing Processing millions of rows when only thousands changed is wasteful. Instead, filter for records created or updated since your last run. Your pipeline will run faster and cost less. 4) Inadequate Error Handling Silent failures mean bad data reaches your dashboards and reports before you notice. Add proper error handling and email alerts so you know immediately when something breaks. 5) Suboptimal Partitioning Partition your tables on columns you actually filter by (like date or region). Aim for partitions between 1-10GB. Don't partition on columns with too many unique values. 6) Poor Orchestration If Job 2 needs data from Job 1, make sure Job 1 finishes first. Databricks workflows handle this for you, no more guessing or manual scheduling. 7) Limited Monitoring Don't wait for users to tell you the data is old or missing. Monitor job performance and data freshness. Get notified automatically when pipelines run late or fail. 8) Inefficient Joins Joining large tables is slow and expensive. If one table is small (under 10GB), broadcast it to all machines. This avoids moving massive amounts of data around. #dataengineering #databricks #cloudcomputing

🚀 Managed vs External Delta Tables in Databricks (Quick Guide) While working with Azure Databricks & Delta Lake, one core concept every Data Engineer must know is: 👉 Managed vs External Tables This directly impacts data safety, governance, and production design. ✅ Managed Table Databricks manages storage No location specified DROP → data + metadata deleted CREATE TABLE managed_tbl (  order_id INT,  cust_id STRING,  salesman_id STRING ) USING DELTA; 📍 Stored in: dbfs:/user/hive/warehouse/ Best for → temp/curated/quick analytics ✅ External Table You control storage (ADLS/S3/Blob) Location specified DROP → only metadata deleted (data safe) CREATE TABLE ext_tbl (  order_id INT,  cust_id STRING,  salesman_id STRING ) USING DELTA LOCATION '/mnt/external/ext_tbl'; Best for → production, bronze/silver layers, governed data 💡 Real-world practice Most projects use: ✅ Bronze/Silver → External ✅ Gold → Managed/External #Azure #AzureDataEngineer #AzureDatabricks #DeltaLake #DataLake #Databricks #ADLSGen2 #ADF #PySpark #remoterole #immediatejoiner #fulltime #SparkSQL #Lakehouse #BigData #DataEngineering #ETL #MicrosoftFabric #CloudData #Analytics #DataPlatform #opentowork

I improved query performance by 30% on Databricks at Goldman Sachs. Here's exactly what we changed (and why most teams miss this): We had a financial analytics pipeline running on Azure Databricks — trading data, risk calculations, P&L aggregations. Queries were slow. Business users were waiting. Leadership was asking questions. We didn't buy more compute. We didn't upgrade the cluster. We fixed the fundamentals. Here's the 5-step playbook: 1/ Stop scanning entire tables Most engineers write SELECT * and move on. We rewrote queries to hit only the partitions needed. Date-based partitioning on transaction tables cut scan volume by ~60%. 2/ Z-ORDER on the columns you filter most On Delta Lake, Z-ORDER co-locates related data on disk. For us: instrument_id and trade_date. Query engines started skipping irrelevant files automatically. 3/ Eliminate shuffle in Spark joins Wide transformations = expensive shuffles. We replaced SortMergeJoins with BroadcastHashJoins wherever the smaller table fit in memory. Execution time dropped significantly on our most complex aggregations. 4/ Cache aggressively — but intentionally We were re-computing the same reference datasets 10+ times per pipeline run. One .cache() call on the right DataFrame. Done. 5/ Use OPTIMIZE + VACUUM regularly Small files are a silent killer in Delta Lake. We ran OPTIMIZE to compact them and VACUUM to purge stale versions. Read performance improved noticeably within a single run. The lesson? Before you scale up infrastructure — audit the queries running on it. 90% of performance problems are logic problems, not resource problems. If you're working with Databricks or Delta Lake, try these before your next cluster resize request. What's the most expensive performance mistake you've seen on a data platform? Drop it in the comments 👇 #DataEngineering #Databricks #ApacheSpark #DeltaLake #Azure #Snowflake #DataPlatform #ETL #BigData #CloudEngineering

Do you want to build advanced observability system for your Databricks platform? You have to check this blog post, with complete guide on how to do it 👇 Key Highlights: • 📊 Queryable telemetry from jobs, pipelines, and clusters in system tables • 🔍 Identify jobs that produce unused data to reduce cost • ⏱ Detect missing timeouts and long runtimes before SLA breaches • 🛡 Spot legacy runtime versions for security and performance • 👥 Pull job owners instantly for faster remediation I've been using the new System Tables for Lakeflow jobs to turn platform telemetry into a single source of truth. Because the tables are read‑only and live in the system catalog, I can write ordinary SQL to pull job configs, task timelines, cost attribution and lineage. With a few joins I can surface jobs that write tables nobody reads, flag runs that exceed expected duration, and list tasks still on deprecated runtimes. The results feed directly into the Lakeflow dashboard template, giving the whole engineering team a shared view of reliability, cost and hygiene. Since everything is queryable, audits and alerts become repeatable queries rather than ad‑hoc scripts. In practice this has reduced the time spent chasing 3 AM alerts and made ownership clear for every pipeline. Which observability signals have helped you catch issues before they impact SLAs? Tutorial - https://lnkd.in/dQwYciy7 #DatabricksAdministration #DataObservability #Lakeflow