Tim Fox

Still treating ETL like a black box? Most teams only find out their data pipeline failed after something breaks downstream. So I built a real-time ETL monitoring dashboard—clean, visual, and actionable. Not just for engineers, but for anyone who needs confidence in their data flow. Here’s what I wanted this to do: ✅ Track success & failure rates over time ✅ Detect API performance drops before stakeholders notice ✅ Monitor data quality with actual validation metrics ✅ Log error patterns for root cause—not guesswork 📊 Data isn’t useful if you can’t trust the pipeline that powers it. This view is part of my analytics portfolio where I share solutions I’ve built across reporting, automation, and performance strategy. https://lnkd.in/gNPWFH5B — 💬 Curious what metrics matter most for your own pipelines? Drop your must-track KPI below—or ask me how I’d build it. #Analytics #ETLMonitoring #DataStrategy #OpsDashboard #BusinessGrowth

325

Big news: Snowflake Ventures is leading a strategic equity financing in AtScale. This investment validates the critical role of semantics in the modern enterprise data stack. As enterprises consolidate data on Snowflake AI Data Cloud, a unified, governed semantic layer is essential for consistent metrics and business logic. A semantic layer like AtScale makes accessing that data more accurate, accessible, and efficient by bridging the gap between business language and data language. AtScale’s Universal Semantic Layer provides this single source of semantic truth for customers like Fidelity Investments and The Home Depot. The company remains independent and interoperable, supporting a broad ecosystem of data platforms, BI tools, and AI applications. This is a good moment to reflect on the future of data and AI. New to semantic layers? I'll post a tutorial link below. And here are five big topics from the 2025 Semantic Layer Summit (link to Substack articles on these below): 1️⃣ Semantic layer standardization is coming 2️⃣ Semantic layers are BI’s Gift to AI 3️⃣ Semantic layers make AI 3X more accurate 4️⃣ Long live SQL (with semantic layers to help) 5️⃣ The great debate: Should semantics be in your database? This is a pivotal moment for the data industry, and I’m thrilled to be a part of it! https://lnkd.in/eVJ276Pp

88

10 Comments

We tested Snowflake's Gen2 warehouses on 2 billion query profiles across 14K warehouses here at Capital One. Here is where we saw improvement: 🍎 Approximately 30% of queries fell into categories that could benefit from Gen2—specifically those involving DELETE, UPDATE or MERGE. 🍎 Long-running DML operations: A significant subset of queries running longer than 30 minutes were performing complex DML tasks. These are prime candidates for Gen2 migration. 🍎 Large scans: Numerous queries scanned over 1 TB of data, where Gen 2’s improved scan performance can provide material runtime reduction. 🍎 Combined heavy patterns: A smaller but critical set of queries both scanned over 1 TB and ran for 30 minutes or more, often involving DML operations—these represent the highest potential gain from Gen2. These findings support a targeted migration strategy: focus on warehouses with a significant concentration of eligible workloads rather than migrating all warehouses indiscriminately. Check out the full blow written by our resident expert and all around awesome guy - Ganesh Bharathan https://lnkd.in/g8XufZaV

292

14 Comments

This is a big feature Snowflake just released. Support for writing data in an external catalog. This means you don't have to use Polaris but could be using something self-hosted, an independent provider or something like AWS Glue. TBC how easy this is to use but as Kyle Onehouse pointed out Snowflake are doing a huge amount of commits into the iceberg repo atm and this interoperability will be a cornerstone of any enterprise serious about using Iceberg for this analytics/warehouse type data Also becomes fundamental to have consolidated orchestration and metadata since you're stitching multiple systems together here and inevitably errors will come and trying to work out where they are will become a huge pain and good news for Yossi Reitblat too Happy Saturday!

123

6 Comments

I’ve seen teams burn $100k a year on Snowflake compute they didn’t realize was running. Don't believe me? $100k isn't even that out of the ordinary. One of my favorite articles, which I read about three years ago now, was about an engineer who accidentally saved 500k by changing a few Snowflake settings. On a similar note, I’ve cut multiple teams data stack costs by 20, 30 and in some cases, 50%....adding up to millions over the last few years. So I can see this happening. Surprisingly, these costs could have often been avoided. Here are the levels of cost saving activties you can take on from easy to hard. Level 1 - Easy Fixes - Warehouse Configuration - If your warehouse default auto-suspended is 10 minutes, set it to 1 - Switch scaling policy to Economy if you can tolerate minor queueing - Right-size your warehouse! If you’re using a medium where a small will do you could easily be spending 2x what you need to spend Level 2 - Medium Fixes - Query Performance - If you’ve got long running queries that are full refreshes, change them to be incremental where possible - Avoid view-on-view-on-views. I see this all the time for dashboards with both gives terrible performance for said dashboard and runs up your bill - Tune your SQL, filter early, select only needed columns - Use cluster keys on large tables to improve pruning. Level 3 - Culture And Tooling - Set up Resource Monitors to alert or auto-suspend warehouses before they blow through your budget - Tag warehouses and queries by team or project, make usage visible and accountable - There are also some tooling changes you might be able to make whether it be switching out your ingestion solution or running your transforms on different compute How have you saved money on your data stack?

109

16 Comments

Uptime doesn't fail due to a lack of dashboards. It fails because someone didn't see the alert. Or didn't correlate the logs. Or didn't escalate fast enough. 🤖 Agentic AI changes this. Imagine: ✅ Reading logs + metrics ✅ Correlating failure patterns ✅ Triggering automated playbooks ✅ Escalating with full context - before PagerDuty ever dings It's not about replacing SREs. It's about giving them a tireless copilot that never blinks. AI won't just report the incident. It'll handle it. #agenticai

226

11 Comments

Model lineage a huge frontier of MLOps right now. There are some *super* exciting things happening in lineage right now. This would have saved my last company $20k/mo on postgres RDS--plus a heck of a lot of other benefits. The DE world has not had an amazing answer for data lineage either. But now we're seeing tools like - OpenMetadata - DataHub - AWS DataZone - Databricks working on "Bring Your Own Lineage" trying to make company-wide lineage a thing. === SQL tables: dbt, sqlmesh, snowflake will show you how your SQL tables derive from each other. They'll even show you "column-level lineage", exactly how each column is computed from upstream columns. Way cool. But life exists before the left-most SQL tables: data sources and ingest pipelines. === Dagster (and now Airflow 3.0 and I think Prefect) can show the dbt stuff AND they can show data sources and assets produced by pipelines, like ML models. But the picture is still incomplete. === The final assets have consumers. Dashboards, powerpoints, agents, full-stack apps, teams, the CEO. === Why does this matter? What if you change a table. How do you know what breaks downstream? What if you make a code change to a data / ML pipeline? What if you WANT to make a change, but don't know how risky it is? Who do you ask / notify? (this is how we were stuck paying for a $20k/mo postgres instance--we had 5-years worth of ML pipelines reading/writing to it) What if you want to build a pipeline? How do you know which upstream assets are refreshed and/or tested regularly? Don't build on brittle, untested, stale data sources. Garbage in, garbage out. === What if something is broken? Who do you ask upstream? === Keep in mind: the graph you see in this image isn't static. A new model version, might use new features, which are built from new SQL queries, which use different upstream tables. You might add new tables and pipelines. === OpenLineage is a lot like OpenTelemetry. It's an open standard for logging lineage events wherever they are generated. Airflow is instrumented with OpenLineage--you can send those events to any "lineage backend" such as AWS DataZone (MWAA does) or OpenMetadata. I believe Metaflow by Outerbounds is exploring something similar. Databricks has some OL support too. Rumor has it they're working to be more of an open OL backend (Bring-your-own-Lineage) I really hope all the DAG tools (both DS and DE) and all the SQL transform tools instrument their code with something like OpenLineage. === I also hope these tools represent non-tabular data soon like text, embeddings, images, ...

77

10 Comments

The last decade of companies showed that the real power is not just in storing data but it is in turning operations into a model the organization can run. Palantir’s contribution was not “better dashboards.” It was the idea that decision-making at scale requires a substrate: one that can hold meaning, operationalize actions, and support “what-if” reasoning. In practice, that substrate shows up as three layers. There’s a semantic layer that makes data legible in the same language across systems. There’s an execution layer that turns that shared language into workflows: real operations that read and write state (and goes beyond just a dashboard from Tableau or Alteryx). And there’s a simulation layer that lets the business ask counterfactuals and propagate constraints through a model of the world. That stack is why Palantir works so well in asset-heavy industries. Healthcare operations, aviation, manufacturing, utilities, domains where the nouns are stable, the constraints are physical, and the consequences of being wrong are expensive. When the model is right, decisions compound. When the model is shared, execution becomes coherent. But there’s also one missing piece or blind spot. Physical-world industries don’t have one ontology, but they actually have two. The first is the explicit ontology, the one we get: the formal entities and relationships you can put in schema. Patients, encounters, diagnoses, procedures. Aircraft (if not's boeing, im not going!) , crews, gates, routes. Assets, substations, feeders, work orders. Production lines, SKUs, batches, inspections. These nouns are stable, well-defined, and capturable. They’re also the part most platforms can connect and normalize. The second is the behavioral ontology: how those entities actually behave together in practice. It’s the unwritten rules and scar tissue that make the difference between “works on paper” and “works in reality.” In a hospital, it’s the charge nurse’s instinct to order the full panel for admits from a specific physician because the workup is predictably incomplete. In an airline ops center, it’s the dispatcher’s rule to avoid pairing a certain captain (who does not want to mentor!) with junior first officers on a redeye because it reliably creates cockpit friction. In manufacturing, it’s the plant’s quiet constraint, run this SKU on Line 3, because Line 2 drifts just enough to crush yield downstream. So, the explicit ontology tells you what exists and PLTR also cracked the "what if" simulations but the behavioral ontology tells you how it really works. This is where most deployments leak value, you can build a world model and still lose the most important part of how work gets done. When the experienced operators leave, the organization loses its edge.... despite having invested millions into platforms that were supposed to preserve it. Why? Because behavioral ontology doesn’t appear automatically. It has to be captured via decision traces!

143

41 Comments

Databricks is about to hit a $100B valuation. Let that sink in. Eight months after raising at $62B, they're signing term sheets north of $100B. But here's what the martech world is missing: This isn't about data warehouses anymore. This is about who owns the future of customer intelligence. Remember when CDPs were going to solve everything? Cute. While we were arguing about identity resolution, Databricks quietly built the infrastructure to make every CDP obsolete. They're processing billions of data points daily with 140% net revenue retention. Their customers aren't just storing data - they're building empires. Here's the spicy take: Braze is about to become exponentially more valuable. Why? Because while everyone else is trying to figure out how to build engagement on top of lakehouses, Braze already has the activation layer every Databricks customer desperately needs. You can have all the AI models in the world, but without Braze's activation engine? You're just sitting on expensive insights. I've seen this firsthand: brilliant data science collecting dust because it couldn't connect to real-world customer interactions. The $37.8B data integration market by 2032? That's just table stakes. The real gold rush is helping enterprises connect their AI-powered data foundations to customer engagement. Every dollar of Databricks software drives $3-5 in partner services. Think of the untapped potential. For Stitch, this convergence isn't a surprise—it's the future we've been building toward. When we launched our Marketing Innovation Practice earlier this year, it was with this exact scenario in mind. We saw that the next frontier wasn't just about helping brands get the most out of Braze, but fundamentally re-architecting how the work of marketing gets done. It’s about making the entire martech ecosystem dance with these powerful new data foundations. Now, we're architecting the definitive bridge between the $100B data platforms and the martech world. Companies don't need another services partner who can configure a campaign. They need a strategic architect who can make Databricks and Braze operate as a single, intelligent organism. The gap in the market is staggering. Databricks has thousands of partners, but they are overwhelmingly data engineers and cloud infrastructure specialists. The number of firms with deep, scar-tissue expertise in the actual discipline of customer engagement and marketing strategy are few and far between. This is our moment to lead. The firms still treating data infrastructure and marketing as separate worlds are building for yesterday. Databricks didn't just build a product; they triggered a fundamental market realignment. The question for every brand and every agency is no longer if this changes everything, but who will architect the future.

149

7 Comments

Snowflake 𝗷𝘂𝘀𝘁 𝗻𝗲𝘂𝘁𝗿𝗮𝗹𝗶𝘇𝗲𝗱 𝘁𝗵𝗲 𝗯𝗶𝗴𝗴𝗲𝘀𝘁 𝗮𝗿𝗴𝘂𝗺𝗲𝗻𝘁 𝗰𝗼𝗺𝗽𝗲𝘁𝗶𝘁𝗼𝗿𝘀 𝘂𝘀𝗲𝗱 𝗮𝗴𝗮𝗶𝗻𝘀𝘁 𝗶𝘁 For years, one objection kept derailing customer workshops and RFP scoring. Databricks repeated it with full confidence. They claimed Snowflake streaming becomes expensive when data arrives in small, constant pieces because the XS warehouse needs to stay alive. Unfortunately, this held up in practice. The pricing model was tied to compute uptime, which turned low-latency ingestion into a budgeting headache. That chapter is closed. Snowflake moved Snowpipe to a simple model based on GB ingested. The cost now tracks the actual workload instead of the arrival pattern. Bursts or trickles behave the same. Early numbers show savings of roughly fifty percent for many teams. 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 This change hits the three pain points that always show up in platform selection. Forecasting becomes easier because the pricing logic is not tied to warehouse behavior. Predictability improves because the ingestion pattern cannot inflate the bill anymore. Justification becomes simpler because the billing unit is consistent from end to end. 𝗧𝗵𝗲 𝗰𝗼𝗺𝗽𝗲𝘁𝗶𝘁𝗶𝘃𝗲 𝗽𝗿𝗲𝘀𝘀𝘂𝗿𝗲 Databricks still brings multiple pricing units to a single pipeline. DBUs for compute, a different unit for SQL, another for apps, and a Serverless layer that feels opaque to many customers. Teams regularly ask why a basic workflow requires several cost models and an Excel sheet to keep track. Snowflake removed the weakest part of its story. Ingestion now behaves the way enterprises expect a cloud platform to behave. Clear. Measurable. Directly tied to value. The platform rivalry will continue, but ingestion economics are no longer a valid reason to hesitate. How long until competitors simplify their own pricing to keep the comparison fair? #Snowflake #DataEngineering #ModernDataStack #Snowpipe #CloudArchitecture #DataPlatforms #CostOptimization #StreamingData #RFP #Databricks

618

20 Comments