Optiver’s Post

Most people still talk about “real-time” like it’s an upgrade. In electronic trading, it’s the baseline. Optiver’s Databricks Apps approach shows where trading infrastructure is heading: moving from siloed, on‑prem systems to a unified Databricks platform, cutting latency from minutes to seconds with Spark Structured Streaming, and giving traders self‑serve Dash dashboards, backed by smart caching, version control, and Unity Catalog for control and reliability. It’s a clear example of live insight at scale, built for impact, not just oversight. Mercedez Lockett-Hart, MBA Jonathan Searle Deb O'Sullivan Joanna Vallejo Andrea DeSosa Junta N. Cyril Cymbler Jennifer Miller Rajaram Suresh

Databricks makes Zerobus Ingest generally available, enabling real-time data streaming into governed Delta tables without intermediate message brokers. #Databricks #ZerobusIngest #DataIngestion #Lakehouse Link to article - https://lnkd.in/esYCPqYG

Hey #Microsoft #DataFactory fans! I just published a quick 6 minute video of my top 3 principals you should follow when operationalizing your Fabric Data Factory pipelines in production environments: 1. Use SPN/WI auth in pipeline activities (where available) 2. Connect to an external Git for CICD and DR 3. Hyper-parameterize your pipelines and use variable libraries for global parameter support, change values through CICD process, and flexible metadata driven pipelines. I hope you enjoy it! https://lnkd.in/gvhiigMc

Day 24: Why "Running it every 5 minutes" is not Real-Time. ⏱️ I often hear: "We do real-time! Our Airflow DAG runs every 15 minutes." That’s not real-time. That’s Micro-Batch. And in fraud detection or personalized offers, 15 minutes is an eternity. True Low-Latency systems require a fundamentally different architecture: Structured Streaming. As a Databricks Champion, I leverage the power of the Delta Lake transaction log to build streaming pipelines that are: Exactly-Once: No duplicate records, guaranteed. Stateful: We can calculate complex aggregations (e.g., "Number of clicks in the last session") in real-time using Watermarking. Unified API: The best part? It’s the same DataFrame API as batch processing. I can test logic in batch and deploy in streaming. Real-World Use Case: I architected a Clickstream analyzer that ingests millions of events via Kafka -> Auto Loader -> Delta Table. The latency went from 1 hour to 10 seconds, allowing the marketing team to trigger "Abandoned Cart" emails instantly. Stop waiting for the batch window. Curious about Streaming architectures? See my event-driven patterns on my hub: 🔗 [Leo’s Digital Hub] https://lnkd.in/gR5sSbv4 #Databricks #DatabricksChampion #Streaming #Kafka #RealTimeData #StructuredStreaming #LeoHuang

One Databricks concept that really stood out to me recently is Unity Catalog. At first glance, it sounds like just another governance layer. But once you understand it, you realize how critical it is in real-world data platforms. Unity Catalog brings everything under one roof — data access, permissions, lineage, and auditing — across multiple workspaces. What I found especially interesting is how it simplifies governance: i. Centralized access control instead of scattered permissions ii. Clear visibility into who is using what data and how iii. Strong alignment with enterprise and compliance needs All of this without moving or duplicating the data itself !! It’s one of those features that quietly does a lot of heavy lifting behind the scenes and makes large-scale data platforms much easier to manage. #Databricks #UnityCatalog #DataEngineering #DataGovernance #DeltaLake #CloudData #LearningJourney

Today I came across a neat Databricks feature: Accidentally dropped a table? With UNDROP TABLE, you can recover dropped Delta tables (and materialised views) in Unity Catalog as long as it’s within the retention window. Limitations to keep in mind: - Materialized views and streaming tables can only be restored if the backing pipeline still exists. If it’s deleted, recovery isn’t possible. - Only works for tables registered in Unity Catalog. - Recovery is only possible within the retention period (default 7 days). One of those features you hope you never need , but are very glad it exists.

Trust me this is the most indispensable feature of Databricks , Even I also got to use this feature few months back 🤠, and it was a huge saviour... #databricks #undrop #table

Microsoft Fabric introduced OneLake catalog federation (Beta) in Azure Databricks Lakehouse Federation, which simplifies multi-engine analytics by enabling Unity Catalog in Azure Databricks to query data stored in OneLake. This allows you to analyze Fabric tables without copying the data. Now, you can query Fabric tables directly from Databricks Unity Catalog without moving a single byte of data. Combine that with Direct Lake mode for Power BI, and we’ve officially entered the "Zero-Copy" era of analytics. Faster insights, lower costs, and one version of the truth.

Subscribe! E2/2026 - This episode of Subscribe! provides a 10 minute overview of Microsoft Fabric Real-Time Intelligence, a unified solution designed to close the digital feedback loop by capturing and acting on data at the speed of business. While traditional real-time pipelines are often cost-prohibitive due to complex integration requirements, this platform simplifies the process for data analysts by layering an opinionated application on top of battle-hardened infrastructure like Event Hubs, Stream Analytics, and Kusto (KQL) Database. We explore how the Real-Time Hub acts as a centralized catalog for managing diverse data streams, from Kafka and AMQP endpoints to Change Data Capture (CDC) from SQL databases, all while maintaining automated schema propagation.

🚀 Unity Catalog Tables vs Delta Tables in Azure Databricks — What’s the Difference? A lot of data engineers get confused between Unity Catalog tables and Delta tables. The key thing to understand is: 👉 They are NOT competitors — they solve different problems. 🔹 Delta tables (built on Delta Lake) focus on storage reliability and performance: ✅ ACID transactions ✅ Schema enforcement ✅ Time travel/versioning ✅ Reliable batch + streaming processing Think: How data is stored safely. 🔹 Unity Catalog tables focus on governance and access control: ✅ Centralized metadata management ✅ Fine-grained permissions ✅ Data lineage tracking ✅ Cross-workspace governance Think: Who can access the data and how. Most modern architectures combine both: Storage → Delta Lake → Unity Catalog governance This gives you performance + security + scalability — exactly what enterprise data platforms need. #AzureDatabricks #DeltaLake #UnityCatalog #DataEngineering #CloudData #BigData #ETL #DataGovernance #Azure #DataArchitecture #AnalyticsEngineering #ModernDataStack