Gilad Lifshitz

When data agents fail, they often fail silently - giving confident-sounding answers that are wrong, and it can be hard to figure out what caused the failure. "Building and Evaluating Data Agents" is a new short course created with Snowflake and taught by Anupam Datta and Josh Reini. This course teaches you to build data agents with comprehensive evaluation built in. Skills you'll gain: - Build reliable LLM data agents using the Goal-Plan-Action framework and runtime evaluations that catch failures mid-execution - Use OpenTelemetry tracing and evaluation infrastructure to diagnose exactly where agents fail and systematically improve performance - Orchestrate multi-step workflows across web search, SQL, and document retrieval in LangGraph-based agents The result: visibility into every step of your agent's reasoning, so if something breaks, you have a systematic approach to fix it. Sign up to get started: https://lnkd.in/gVwKuCCQ

6,625

224 Comments

My current workflow developing on #Databricks 👇 - Connect to Databricks via Databricks VS code extension. - Develop in VS code with interactive cells. - Use uv for setting up virtual environment, package building (build backend setuptools, but can be hatchling) and dependency resolution. - Start writing modular code & package it as soon as possible keeping the scripts clean. - If some code can’t be executed from the VS code, sync the files (including wheel files) using VS code extension and execute in a notebook (do not forget to install the package). - When the script is ready, use asset bundle deploy & run command to execute it on interactive cluster. - Databricks.yml definition contains artifacts section that will build and upload package to the bundle location and make it accessible for the script to run. - It is important to set dynamic_version=True when running on interactive cluster. In this way, all package changes will be availble on the cluster. Here is an example of basic databricks.yml file 👇.

576

37 Comments

🚀 Big news from Snowflake: Workspaces is now GA — bringing a modern SQL development experience directly into Snowsight! Some highlights I’m excited about: -- 🗂️ Organize projects with file-based workflows, nested folders, and easy conversion from Worksheets -- ✨ A powerful editor with tabbed layout, split panes, pinned outputs, and faster performance -- 📊 Built-in charting & column stats for instant data visualization -- 🤖 Snowflake Copilot (inline, public preview) to generate, edit, and explain SQL on the fly -- 🔗 Native Git integration: pull, push, branch, and resolve conflicts right inside Snowsight This is a huge step forward in making Snowflake not just the best Data & AI platform, but also a first-class development environment. 👉 Check out the details here: https://lnkd.in/gnJXHDVM

1,045

18 Comments

Expensive platforms like Databricks and Snowflake aren’t the way in data engineering. You can build your stack entirely open source as well!

575

54 Comments

87% of modern data teams are using dbt today. But most professionals are still stuck in outdated workflows. If any of this sounds familiar... - You’re still writing complex SQL transformations manually - You spend hours debugging broken pipelines - Your models live in spreadsheets or tribal knowledge - You don’t fully understand version control, testing, or documentation - You’re applying for roles that ask for DBT, and skipping them because you don’t feel ready Then this bootcamp is for you! The 𝐃𝐚𝐭𝐚𝐆𝐲𝐦 Bootcamp: Train. Build. Transform. ✅ 𝐓𝐫𝐚𝐢𝐧: World-class instruction by Bruno Lima, dbt Community Award winner (2023 & 2024), Tech Lead at phData, speaker at Coalesce, and one of LATAM’s top dbt educators. ✅ 𝐁𝐮𝐢𝐥𝐝: Hands-on projects designed to simulate real-world challenges—so you learn by doing, not just watching. ✅ 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦: DBT is now a requirement in many data roles. This bootcamp helps you close that gap and become job-ready faster. 𝐖𝐡𝐲 𝐥𝐞𝐚𝐫𝐧 𝐃𝐁𝐓 𝐧𝐨𝐰? - It’s the industry standard for modern data transformation - DBT roles pay ~20–30% more than traditional analyst roles - It's SQL-friendly—no need to learn a new language - It’s built for collaboration, scalability, and testing - Used by Netflix, Meta, Spotify, Canva, and 30,000+ companies globally Whether you're transitioning into analytics engineering or just want to level up as a data analyst, this bootcamp gives you the skills that hiring managers actually look for. 💡 I’m also attaching an exclusive DBT cheat sheet by Albert, perfect for quick wins and pro tips. 📥 𝐄𝐧𝐫𝐨𝐥𝐥 𝐡𝐞𝐫𝐞: https://lnkd.in/daEEvi-f 💸 Use code "𝐕𝐄𝐍𝐊𝐀𝐓𝐀" for 25% off ♻️ Save it for later or share it with someone who might find it helpful! 𝐏.𝐒. I share job search tips and insights on data analytics & data science in my free newsletter. Join 14,000+ readers here → https://lnkd.in/dUfe4Ac6

324

47 Comments

Your competitor made >$10M with AI predictive model. But your data team delivered 221 dashboards in 12-months. Your team not slow, just broken. Been hearing this for the past weeks “Our data team is so slow.” Iterations, requests take weeks. Priorities keep shifting. Everyone’s busy, but nothing moves. Let’s be clear, that’s not a talent issue. It’s a structural failure. Your data team isn’t drowning in work. They’re drowning in dependencies, misaligned ownership, and bad incentives. Mistake #1: You Built by Function, Not by Outcome Most teams look like this: Engineers ingest data Analytics engineers model it BI developers visualize it Analysts most times nobody knows what they doing. Looks efficient on paper. In reality? It’s a garbage. Each role optimizes for its piece, not the outcome. The engineer ships a pipeline which already made. The analyst ships a dashboard which usless. Nobody owns the impact that dashboard is meant to enable. That’s why velocity dies. Fix: Organize around data products, not job titles. Each data product (e.g., revenue insights, supply chain metrics, marketing attribution) should have cross-functional memeber that owns it end-to-end. When the same squad owns ingestion → modeling → delivery, you eliminate 80% of the handoffs that kill speed. Mistake #2: You Measure Output Instead of Impact You’re tracking the wrong KPIs. Executives love vanity metrics: “How many dashboards did we build?” “How many pipelines went live to prod?” That’s noise. What you should measure is throughput, how fast the business goes from question, problem → trusted data → impact If your don't have clera defintions of done, ETA and ROI. You doing nothing. Fix: Velocity isn’t how fast engineers code, it’s how fast the business gets results. Mistake #3: You Built a Platform Without an Operating Model Every company says they have a “data platform.” Few have a data operating model, a clear system for ownership, collaboration, and accountability. So you get chaos: Five teams writing pipelines differently Conflicting metrics definitions Endless meetings to resolve “which number is right” That’s not a tooling problem. It’s an org-design problem. Fix: Define how your platform and data teams interact. Who owns ingestion? Who owns definitions? Who approves changes? And for every dataset, model, and 'data product', name an owner. Velocity doesn’t come from adding tools. It comes from removing friction. Your team isn’t slow because they lack skills. They’re slow because you built an org that rewards activity over impact. You don’t fix that with more headcount or another vendor. You fix it by rewiring focusing on impact, trust and adoption. One team, one outcome, 90-day delivery cycles. If your team’s buried under tickets and “urgent requests,” it’s not a workload problem, it’s an operating model problem. 🧬 Repost if you think most data teams creating bureaucracy.

373

37 Comments

The real reason your data team is underperforming? You're using them like a dashboard concierge service. And treating strategy like a synonym for “data stack.” Here’s how most companies use their data team: → To build dashboards no one uses → To build dbt models that collect dust → To chase tool-of-the-month hype → To enforce rules no one understands And leaders wonder why the data team feels burned out and the business still isn’t “data-driven.” Data teams are often treated like: - Internal service desks taking tickets - Janitors cleaning up tech debt - Compliance cops for naming conventions But value comes when you flip the model. Less output, more outcomes. Less dashboards, more decisions. Less data police, more collaboration. Your job as a data leader isn’t to crank out reports faster. It’s to stop the madness and ask: What are we even doing this for? ♻️ Repost if your dbt repo has more models than your stakeholders have questions 👉 And join 3,000+ data leaders who read my free newsletter for weekly tips on building impactful data teams in the AI-era: https://lnkd.in/gHMYtWxE

1,931

73 Comments

𝗪𝗵𝗮𝘁’𝘀 𝗡𝗲𝘄 𝗶𝗻 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗔𝗜/𝗕𝗜 𝗶𝗻 𝗢𝗰𝘁𝗼𝗯𝗲𝗿 𝟮𝟬𝟮𝟱 Databricks just dropped a huge update for AI/BI Dashboards and Genie, making analytics smarter, faster, and easier to share. 𝗘𝗺𝗯𝗲𝗱 𝗱𝗮𝘀𝗵𝗯𝗼𝗮𝗿𝗱𝘀 𝗲𝘅𝘁𝗲𝗿𝗻𝗮𝗹𝗹𝘆: share secure, governed insights with customers and partners (no Databricks login needed). 𝗦𝗹𝗮𝗰𝗸 𝗰𝗵𝗮𝗻𝗻𝗲𝗹 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗽𝘁𝗶𝗼𝗻𝘀: deliver live dashboard snapshots straight into your team’s workspace. 𝟭𝟳𝟬+ 𝗻𝗲𝘄 𝗰𝘂𝘀𝘁𝗼𝗺-𝗰𝗮𝗹𝗰𝘂𝗹𝗮𝘁𝗶𝗼𝗻 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀: richer metrics, easier logic. 𝗡𝗲𝘄 𝘃𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀: funnel and waterfall charts for better storytelling. 𝗚𝗲𝗻𝗶𝗲 𝘂𝗽𝗴𝗿𝗮𝗱𝗲𝘀: expanded APIs, better benchmarking, a smarter Knowledge Store, and transparent “thinking steps.” 𝗚𝗼𝘃𝗲𝗿𝗻𝗲𝗱 𝘁𝗮𝗴𝘀 & 𝗰𝗲𝗿𝘁𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻: apply Unity Catalog tags to dashboards and Genie spaces for trust and discoverability. Databricks AI/BI keeps unifying visualization, natural-language exploration, and governance helping every team turn data into decisions. 🔗 Read the full roundup: 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗕𝗹𝗼𝗴 𝗪𝗵𝗮𝘁’𝘀 𝗡𝗲𝘄 𝗶𝗻 𝗔𝗜/𝗕𝗜 (𝗢𝗰𝘁 𝟮𝟬𝟮𝟱): https://lnkd.in/d74HBhqJ #Databricks #AIBI #Genie #DatabricksMVP

349

10 Comments

🚀 Big news for Databricks Assistant users — AI just got way better at editing your code. Introducing Assistant Edit Mode 🧠💻 — a new way to apply AI-generated suggestions across multiple cells at once. Now you can: ✅ Refactor logic, rename variables, and clean up code — all with a single prompt ✅ Migrate SQL, Pandas, or Python code faster and more accurately ✅ Standardize style, remove clutter, and even generate test scaffolding automatically No more tedious, cell-by-cell edits. Edit Mode understands your entire notebook context — so your changes are consistent, fast, and easy to review. In early testing, it cut refactoring time by over 50%. This is just the start. You will soon hear about Edit Mode in AI/BI Dashboards and how we are integrating agent-like workflows into Databricks Assistant. Stay tuned! 📖 Read the full blog and see it in action 👇 #Databricks #GenerativeAI #DataScience #MachineLearning #DataEngineering #Productivity #AI

8

Don’t learn Databricks. At least not yet. Before Databricks, learn SQL. Because every data problem eventually becomes a query. Learn Python. Because pipelines, transformations, automation, and ML all depend on it. Learn Spark. Because Databricks is not magic. It is Spark with a managed platform. Databricks does not teach you fundamentals. It only exposes them. If you jump straight to Databricks without understanding SQL execution, Python logic, or how Spark actually works, you will struggle the moment something breaks. Buttons won’t save you. Notebooks won’t save you. Auto-scaling won’t save you. But if you know SQL, Python, and Spark, Databricks becomes powerful instead of confusing. Platforms change. Foundations don’t. So don’t rush to learn Databricks. Build the basics first.

625

24 Comments

"How do you define messy data?" Well... - Dates stored as text IDs duplicated, re-used, and occasionally replaced with emojis - Nulls labeled as N/A, NA, none, and ¯\_(ツ)_/¯ - Source File is a scanned PDF (in Japanese), loaded sideways, hosted on SharePoint, and the only admin left the company three years ago

110

36 Comments

If you're still writing transformation logic buried deep in SQL scripts… it's time to meet your new best friend: DBT. Data Build Tool (DBT) is changing how modern data teams operate. It’s not just about SQL — it’s about:   Version-controlled, testable, modular transformations   ELT done the right way: Raw → Trusted → Transformed   CI/CD, data lineage, and JINJA macros out-of-the-box   Lightweight workflows without heavyweight orchestration tools No more: 🔁 Copy-pasting SQL logic 🤯 Wondering where the data came from ❌ Manually debugging failed queries at 2AM Instead, imagine: ✅ Easily testable models ✅ Documentation that writes itself ✅ Analysts owning the transformation layer — with zero compromise on engineering principles This PDF by Praveen walks through:   DBT core concepts   Real project flow (from raw files & cloud DBs)   Materializations: tables, views, incremental, ephemeral   How to build, test, and deploy transformations with just a few commands If you're serious about modern data engineering, DBT isn’t optional — it’s essential. Let this be your first step in mastering it. If you've read so far, do LIKE the post 👍   𝐏.𝐒: Get a HIGH Paying "AWS" Cloud Data Engineering Job: https://aws.sachin.cloud & Learn no-nonsense "AWS" Cloud Data Engineering: 𝐖𝐡𝐚𝐭𝐬𝐚𝐩𝐩 group for notification : https://lnkd.in/eyz5sQtN 

174

24 Comments

Most aspiring Data Analysts fail interviews… not because they aren’t smart, but because they don’t know what they’re walking into. Here’s the problem: ❌ You prepare only for SQL… and then get blindsided with a business case study. ❌ You focus on charts… but can’t explain why your analysis matters to the business. ❌ You tell your story… but without structure, it falls flat. The truth? There are 5 types of interviews you must be ready for — and each tests something different. The good news? ✅ Once you know what to expect (and how to prepare), the stress disappears. ✅ You’ll walk in confident, structured, and ready to impress. That’s exactly what I help my students do, and why they’re landing jobs in just 3–6 months. 🚀 Want my step-by-step system to prepare for every interview type? 👉 Join my free training here: https://lnkd.in/ekp-jmD2

38

7 Comments

I used to think the answer to data chaos was more dashboards. - more charts. - real-time updates. - cleaner visualizations. Turns out I was solving the wrong problem. Here’s what actually happens on most teams: → they open 3 reports to answer 1 question. → spend 20 minutes interpreting charts. → debate whether the numbers are right. The problem isn’t dashboards. It’s that business users are forced to interpret data at all. They should be able to ask: “What is our average sales cycle length?" And just get a clear answer. Not a 20-page report with a maze of filters in 6 different colours. AI will change this. Soon, everyone will make data-driven decisions, in seconds. But only if the data foundation is solid. Data science was the hottest job in 2020. In the next decade, it’ll be data engineering.

31

12 Comments

💡 Still running your DWH on SQL Server? Time to rethink. If you have: • a team that kills sessions because of locks • ETL jobs only at night so nothing breaks • schema changes that take a week to “coordinate” • fear of DELETE because it might freeze everything …then you’re solving modern problems with outdated tools. Try Apache Iceberg: • ACID transactions • real schema evolution • deletes, merges, time travel • works with Spark, Trino, Flink, Snowflake & more ☁️ In the cloud, it’s faster and way cheaper. Store once in S3/GCS, query with anything. Compute when you need it — no lock-in, no waste. If your “data platform” still looks like SQL Server + nightly jobs + manual babysitting — something’s off. #Iceberg #DataEngineering #ModernDataStack #SQLServer #CloudData #DWH

347

143 Comments