Skipping Bronze layer in data pipelines: dbt + Snowflake benefits

𝐞𝐱𝐭𝐞𝐫𝐧𝐚𝐥 𝐞𝐧𝐠𝐢𝐧𝐞𝐬 𝐜𝐚𝐧 𝐧𝐨𝐰 𝐰𝐫𝐢𝐭𝐞 𝐭𝐨 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞’𝐬 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐭𝐚𝐛𝐥𝐞𝐬 (In public Preview) : Snowflake just announced a major leap forward in open data architecture — 𝐟𝐮𝐥𝐥 𝐛𝐢𝐝𝐢𝐫𝐞𝐜𝐭𝐢𝐨𝐧𝐚𝐥 𝐢𝐧𝐭𝐞𝐫𝐨𝐩𝐞𝐫𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐟𝐨𝐫 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐭𝐚𝐛𝐥𝐞𝐬 𝐯𝐢𝐚 𝐭𝐡𝐞 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐇𝐨𝐫𝐢𝐳𝐨𝐧 𝐂𝐚𝐭𝐚𝐥𝐨𝐠. This means teams can now securely write to Snowflake‑managed Iceberg tables from external engines like Spark, Trino, and others, completing the last mile in truly open, multi‑engine data ecosystems. Until now, external engines could only read from Snowflake’s Iceberg tables. With this update, organizations can maintain a single governed copy of Iceberg data in Snowflake while executing workloads wherever it makes the most sense. It eliminates the headache of multiple copies. #snowflakenewfeatures #datalake #snowflake

If you've ever scratched your head over how to track when rows were created or updated in Snowflake - Minal's got you covered. 🙌 She shared three practical ways to implement row timestamps, and it's exactly the kind of clear, no-fluff content this community needs more of. Whether you're a beginner or just looking for a cleaner approach, this post is worth saving. Go read and give it a like 👇

Medallion architecture is one of those patterns that makes perfect sense until real data shows up. I wrote a blog series that follows a data pipeline from "this is simple" to "why do we have eight transformation functions and whose idea was this" to an architectural pivot that makes the whole thing schema-agnostic. New source? Add rows to a mapping table with no deployment required. The series then follows the data into dbt silver transformations and synthetic test fixtures that capture realistic chaos while masking sensitive data. Then comes the finale that will haunt you: a source submitting duplicate barcodes across two files, every test passing, the pipeline completely unaware. The whole stack is Databricks Community Edition and dbt Core, so free to run yourself. The patterns translate to Snowflake, BigQuery, or any other data warehouse environment. Full series in the comments. #dataengineering #databricks #dbt #monitoring

Choosing between BigQuery, Snowflake, Redshift, or Synapse isn’t about brand preference; it’s an architecture decision. It’s about: • Workload profile • Concurrency needs • Data complexity • Budget elasticity • Governance requirements We’ve seen organizations overspend 30–40% simply because architecture decisions weren’t workload-aligned. The wrong data platform compounds over time. Here’s a comparative analysis for technical leaders 🔗 https://bit.ly/3OeTK9K

Built a Databricks Delta Live Tables pipeline using SQL and Medallion Architecture (Bronze → Silver → Gold) for sales analytics. The project ingests customer, product, and sales source tables, applies data quality checks, performs silver-layer enrichment, and creates gold KPI marts for downstream BI reporting. Key things implemented: SQL-based DLT pipeline Bronze, Silver, Gold transformations Expectations-based quality checks Incremental processing Customer and product enrichment Sales KPI marts DLT lineage DAG This project helped me strengthen my understanding of production-style lakehouse ETL design and how business-facing marts are built in Databricks. GitHub README includes architecture, DAG flow, and project structure. Repo link to the project: https://lnkd.in/diT8XQty #DataEngineering #Databricks #DeltaLiveTables #SQL #ETL #DataEngineer #GitHub #Lakehouse

I just finished architecting an automated, containerized Medallion Data Lakehouse, and I decided to put it to the ultimate test: a complete "Nuke & Rebuild." 💥 In this video, I completely destroy my Snowflake environment and let my orchestration pipeline rebuild the entire infrastructure from scratch in under 3 minutes. The Tech Stack & Specs: 🔹 Orchestration: Dockerized Apache Airflow 🔹 Data Warehouse: Snowflake (Medallion Architecture + RBAC Security) 🔹 Transformation: dbt Core 🔹 Scale: Automated extraction and processing of 100,000+ raw retail records. 🔹 Business Value: Consolidated 8 highly normalized operational tables into a Kimball Star Schema optimized for downstream BI tools like Tableau. Zero data loss. 100% automated recovery. Check out the video to see the DAG in action, and feel free to review the full repository in the first comment! 👇 I'd love to hear from the data community. How do you personally handle testing for silent failures in your ingestion pipelines? #DataEngineering #Airflow #Snowflake #dbt #Docker

Snowflake - UK National Data Library: Distributed Architecture for Research The data landscape for research and the public sector in the UK today is fragmented into thousands of disconnected databases. Snowflake's vision is to establish a National Data Library (NDL) using a distributed architecture that drives growth and improves public services. https://lnkd.in/e6ihFesr #snowflake #datadelivery #nationaldatalibray

Snowflake makes data engineering feel effortless - until you start scaling. One thing I’ve learned working with Snowflake at enterprise scale is this: architecture matters more than warehouse size. Here’s what actually moves the needle: • Snowpipe for clean, low-latency ingestion without orchestration overhead • Snowpark for complex transformations beyond SQL • External tables + streams for predictable incremental processing • Clear modeling layers to avoid expensive, tangled query paths • Monitoring query history to catch silent credit burners early When these pieces work together, Snowflake becomes more than a warehouse. It becomes a full data platform. #DataEngineering #Snowflake #Snowpipe #Snowpark #DataArchitecture #CloudComputing #ETL #DataModeling

Apache Iceberg is quickly becoming the open table standard for modern lakehouse architectures, and Snowflake is meeting it head on.🚀 This blog breaks down when to choose Snowflake managed Iceberg tables, when externally managed tables make more sense, and where Snowflake Open Catalog fits in your data strategy.📊 You will see how to keep catalogs in sync, manage table state, and enable engine agnostic analytics without sacrificing governance or performance. If you want a practical guide that turns Iceberg + Snowflake from buzzwords into an actionable architecture, this is for you. Read the full blog on our website to dive in. 👉 https://lnkd.in/gdqgHN-6