John Hupperts

With Snowflake, Zayo Group centralizes and enriches data from various sources to provide actionable insights for network monitoring, reducing alert volumes by 15%. Watch to learn more about how Zayo leverages Snowflake and DAS 42 to stay ahead in the communications infrastructure industry.

120

2 Comments

Engineering at Snowflake means solving problems no one has tackled before. For Principal Software Engineer Hyungtae Kim and the Cortex team, that meant designing GPU-powered infrastructure flexible enough to handle any AI workload, at scale. Get a behind-the-scenes look at how the foundation was built, what makes it different, and how it’s powering the next era of GenAI: https://bit.ly/4kKjGVI

150

2 Comments

Managing complex data workflows is a challenge, especially when the stakes are high. In this episode, we sit down with Jennifer Melot from the Center for Security and Emerging Technology to explore how her team orchestrates large-scale data pipelines for emerging technology research. We discuss how dynamic DAGs, Airflow automation and scalable infrastructure streamline their data operations and enhance policy analysis. Listen now via the link in the comments below. #AI #Automation #Airflow #MachineLearning

11

1 Comment

Join us as we dive deep into accelerating your data engineering workloads using RAPIDS Accelerator for Apache Spark on AWS. In this episode, we'll explore how to leverage GPU acceleration to dramatically speed up your Spark applications while reducing costs. (edited)

307

76 Comments

Developers build faster when context lives where they work. In this demo, Gene Arnold show's how connecting Atlan’s MCP Server to Cursor brings your data catalog directly into the coding environment- cutting context switching and making data-driven development seamless. This is a simple setup with big impact: instant access to metadata, right where you need it 👇

42

What if I told you that a 1 GiB/s Kafka topic streamed directly into your S3 data lakehouse as an Iceberg table could cost you... $10/hr?  Bufstream does it. It's literally too good to be true. They are a core part of the massively deflationary wave that's exerting commoditization pressure and crushing Kafka margins. How did this Protobuf start up build their own diskless Kafka implementation in no time? 🤨 I spent 20 hours researching them. Here's their story 🧵 

200

19 Comments

Imagine trusting your data to a fax machine.​ That’s the reality for 86% of companies still running core apps on legacy systems.​ Say goodbye to legacy and hello to trusted, AI-ready data. Explore the data modernization guide: https://slalom.ws/3IBY1kX

31

1 Comment

If your team is building with LLMs, you know the value of clean, retrievable data. That's why we're teaming up with Unstructured to streamline one of the hardest parts of AI development. This integration removes friction across the AI stack—from unstructured ingestion to semantic retrieval—so your teams can focus on building. Read more from our Product Marketing Managers, Rini Vasan and Jim Allen Wallace, and Unstructured’s Head of Developer Relations, Maria Khalusova here: https://lnkd.in/e98QqXAB

43

Snowflake Openflow has been costing me $10/day to sit idle with all run times suspended. The Snowflake managed "deployment" compute pools can't be suspended and don't show with the `show compute pools` command. They bill under Cloud Services, not compute pools from what I'm seeing. $3,600 per year won't break the bank for an enterprise, but don't do some experimentation and just leave that thing on!

109

21 Comments

Your data catalog is just the beginning. Forrester Research explains how metadata-driven architecture goes far beyond lineage and sources-it creates a connected knowledge graph of your entire ecosystem. This architecture enables real-time insights, smarter automation, and faster model deployment. Explore the architecture that powers true DaaS success. 📥 Read the full report: https://ow.ly/cOKt50VG4zy

33

1 Comment

System Tables are powerful. They provide unmatched visibility into costs, performance, and usage across your Databricks environment. But here's the catch... visibility alone doesn't optimize anything. That's where Unravel comes in. We transform raw System Table data into intelligent, AI-driven actions, bridging the gap between knowing what's happening and automatically fixing what needs to change. 💡 Move beyond dashboards. ⚡️ Embrace proactive optimization. 🔗 Read the full blog: https://lnkd.in/gcN5EJHZ #Databricks #DataObservability #FinOps #AIOptimization #CloudCosts

6

"bauplan is a lot easier to work with than rolling your own Apache Iceberg setup" "But exactly *how* easy?", you may ask. The answer is 10 lines of code below: pure Python inside your medallion transformations, pure Python when you script your #lakehouse through APIs and typed objects. I met George online a while ago because he found our reference implementation for #PyIceberg+Dremio, and he then re-built the exact same project now using bauplan (and Mage for orchestration). The result? "Compared to my previous setup, Bauplan has been a total game-changer. It drastically simplifies development and cuts back on all the custom integration work I used to do." Crucially, every lakehouse entity (not just the SQL ones!) is "managed under one roof: branches, Iceberg tables, transformations — you name it". Check Georgios (George) Zefkilis full blog post here: https://lnkd.in/e4RYtEEN If you want to run Georgios (George) Zefkilis's example on our free sandbox, ask for an invite: https://lnkd.in/eJq9_XYj

34

12 Comments

Scaling a database shouldn’t mean complexity, downtime, or vendor lock-in. That’s why we built a side-by-side guide comparing CockroachDB and Oracle Globally Distributed Database (GDD) highlighting the ways CockroachDB delivers on distributed SQL in a more resilient, flexible, and simple way than Oracle GDD. 🗳️ Download the guide: https://cockroa.ch/4noA5AF

60

🚀 Build Your Data Lakehouse Instantly from AWS S3 Before analytics, data applications, or AI workflow, there’s a critical first step: data ingestion. Without reliable data ingestion, even the most ambitious data initiatives stall. That’s why we’re excited to announce the launch of File Ingestion in Starburst Galaxy, designed to make building an Iceberg data lakehouse from Amazon Web Services (AWS) #S3 faster and easier than ever. Starburst already supports streaming ingestion from Kafka sources. Now, we’re adding file ingestion, enabling data engineers to instantly hydrate Iceberg tables from AWS S3—without complex pipeline tools or custom code. Once ingested, the data is landed into Iceberg Live Tables, allowing for seamless data management and optimization across your data pipeline. Want to know more? Read the blog post by Ahmed Niyaz, Nick Kessler, and Evan Smith: https://okt.to/1KQ4Vo #DataEngineering #ApacheIceberg #StarburstGalaxy #AWS #DataLakehouse #DataIngestion #ModernDataStack

47

2 Comments

Spent some time yesterday diving into RAG (Retrieval-Augmented Generation) and found these videos to be extremely helpful in understanding how to move from toy projects to production-grade pipelines. If you're working on RAG applications, these are must-watch: 1️⃣ Jerry Liu @ AI Engineer Summit – Talks about challenges in productionizing RAG, evaluation techniques, and how to make retrieval better. Link: https://lnkd.in/gRK8hi2R 2️⃣ LlamaIndex x Arize – Practical tips on fine-tuning embeddings, chunking, and query routing for more robust RAG systems. Link: https://lnkd.in/gdP7RZvW 3️⃣ Langchain RAG Course – End-to-end hands-on tutorial to build a multi-user chatbot using Langchain, FastAPI & Streamlit. Link: https://lnkd.in/gV_e7KGa Definitely worth a watch if you're building with LLMs in the real world.

52

1 Comment

Databricks set a new direction at the 2025 Data + AI Summit. The new mission is clear: build and deploy full-scale data and AI products on a unified, AI-native platform. Here’s what’s new: • Lakebase – Real-time workloads, fully integrated • Agent Bricks – Build, deploy, and manage AI agents • Lakeflow + Declarative Pipelines – From pipeline code to pipeline intent • Lakebridge – AI-powered onboarding, accelerated • Unity Catalog Metrics – Governed business metrics, centralized • AI/BI – Business intelligence with an AI-native interface • Databricks Apps – Serverless, data-native applications • Databricks One – Unified view of metrics, apps, and agents • Free Edition – No-cost access for small teams and prototyping We broke down what each launch means, and how they’re changing the way teams execute AI at scale. Get the details: https://hubs.li/Q03sMykq0 #DataAISummit #Databricks #Indicium #AIExecution #IndiMesh #ModernDataStack

44