Ariel Li

Just completed an instructor-led Snowflake GenAI Training. Here are some valuable insights: 🧠 Cortex AI Functions: Snowflake's built-in AI capabilities allow you to run sentiment analysis, text summarization, and language translation directly within your data warehouse using simple SQL commands like SNOWFLAKE.CORTEX.SENTIMENT() and SNOWFLAKE.CORTEX.SUMMARIZE(). 🔍 Cortex Search: Semantic search that understands context through vector embeddings. We worked with travel agency data to build search services that can find relevant documents based on meaning, not just keyword matching - perfect for knowledge bases and document repositories. 📊 Cortex Analyst: Natural language querying powered by semantic models defined in YAML files. We built a complete travel booking semantic model that lets business users ask questions like "What were our top destinations last quarter?" and get accurate SQL results automatically. 📄 Document AI: Extract and analyze unstructured data from PDFs, images, and documents at scale. Practical Applications We Explored: ✅ Travel Industry Use Case: Built a complete semantic layer for travel bookings with destinations, customers, and revenue metrics ✅ Hands-on coding with cortex-llm-functions and cortex-search implementations ✅ YAML Semantic Models: Learned how to define business logic that translates natural language to accurate SQL queries ✅ Vector Search Integration: Set up semantic search services that understand document context and relationships If you're working with large datasets or looking to make your organization more data-driven, these tools offer compelling advantages with minimal setup overhead. Who else is exploring GenAI in their data stack? Would love to hear your experiences! 👇 #Snowflake #GenAI #DataAnalytics #ArtificialIntelligence #DataScience #MachineLearning #BusinessIntelligence

28

1 Comment

I feel like there should be a bar exam for data scientists. You take it once and then you can forever skip technical assessments.

3

I uncovered $430,000 in untapped revenue using Python and Power BI on the Online Retail UCI dataset, a public dataset from a UK-based online gift retailer. The numbers: $8.74 million total revenue 4,335 customers $474.81 average order value. A simple RFM model grouped customers into five clear segments: Highest Value (80% of revenue) At-Risk Valuables (8%) Sleeping Potentials (5%) Loyal Lost & New. The hidden opportunity? Sleeping Potentials, customers who spent big in the past but haven’t purchased in over 250 days. Though only 5% of the base, they represent $430,000 in potential revenue with just a small nudge. Geography added clarity: United States: $7 million (80%) Netherlands: $300K EIRE: $260K Germany: $200K France: $180K A clean Power BI dashboard with country maps, segment breakdowns and interactive filters made the story instantly clear to anyone. This answered four key questions: Who drives the most revenue? Who are the “sleeping giants” we’re missing? Where should we focus geographically? What’s the real opportunity in bringing customers back? Recommendations: Send a “We Miss You” email to Sleeping Potentials with a 15% discount and free shipping Launch a VIP loyalty program for Highest Value customers to protect 80% of revenue Localize payment options and promotions in Germany and France to grow mid-tier segments.  #DataAnalytics #CustomerSegmentation #RFMAnalysis #PowerBI #Python #RetailAnalytics

16

1 Comment

In my recent conversations, I've noticed a 2 concepts that seem to be causing some confusion in the DS community, and I'd like to take a moment to clarify them: 1. Difference in Bias of an estimator and Bias variance trade off. 2. Covariate shift in causal inferencing vs feature drift. Bias in the Estimator refers to a single, standalone property of a statistical model's prediction. An estimator is considered unbiased if, on average, it hits the true value of the parameter it's trying to estimate. In other words, if you were to repeat your experiment many times, the average of your predictions would equal the true value. Whereas, The Bias-Variance Trade-off is a concept that describes a fundamental tension in model building for choosing the complexity level. Covariate shift in causal inference and feature drift in machine learning are related concepts, but they refer to different problems and appear in different contexts. Covariate Shift in Causal Inference describes a situation where the distribution of the input variables (covariates) in the test data is different from the distribution in the training data, while the relationship between the covariates and the outcome variable remains the same. In a causal context, this is a big problem because causation is lost which Invalidates Causal Conclusions. Feature drift, or concept drift, is a broader term in machine learning. It refers to the problem where the statistical properties of the target variable or input features change over time in an unexpected way.  Hope this help to the readers.

43

1 Comment

Last week, I attended the Databricks Data & AI Summit and passed the Data Engineering Associate certification exam! I enjoyed connecting with fellow data professionals, attending breakout sessions, and learning best practices. Here are three takeaways from the conference that resonated with me: 💡 Migration Strategy: Prioritize Business Impact. When piloting a new product/feature or considering a migration, focus on small use cases and user needs over flashy features. Chintan Patel shared how ThredUp followed this approach during a data warehouse migration to Databricks that empowered users with self-service data tools and enabled data engineering teams to focus on strategic work. 💡 Prevention > Reaction in Data Quality. Data quality and governance should be embedded into every data pipeline stage. Xiaoyu Zhou emphasized a preventative approach to maintaining data quality and provided an invaluable breakdown of automated data quality tools at the ingestion and loading stages. I’m looking forward to trying out the DQX package to monitor data quality for our work at CDPH. 💡 GenAI + SQL: Powerful but Proceed with Caution. Databricks AI Functions lets you query unstructured data using LLMs through simple 'ai_query' calls in SQL. While this is game-changing, it’s our responsibility as the end user to validate and test the output data. The conference reinforced that great data engineering isn't about adopting every new tool, but choosing the right solutions and implementing them thoughtfully. Thanks to Dillon Taylor and Navneet Gupta for the opportunity to attend. If you attended or have thoughts on these topics, let's connect! #DataAISummit #DataEngineering #Databricks #DataQuality #GenAI

42

3 Comments

I just open-sourced TransformDash, a data transformation platform I built after after working with custom Rust transformations, dbt, Airflow, Airbyte, Superset, and Tableau. It aims to bring transformation, orchestration, and visualization into a single tool.​ What it does: - SQL and Python models with Jinja templating - ML model training and inference (e.g., churn prediction, segmentation) - DAG-based dependency resolution and cron-style scheduling - Built-in dashboards and visualizations - Runs on your existing PostgreSQL database (no separate warehouse) It’s built with Python (FastAPI backend) and vanilla JavaScript. Auth uses JWT with role-based access control, and SQL is safely constructed with psycopg2’s composition utilities to avoid injection issues.​ Next step: I’m now working on making it available as a pip-installable package for easier adoption. Repo: https://lnkd.in/e4Y6h92t Feedback and suggestions are very welcome. 😊 #DataEngineering #OpenSource #Python #dbt #DataTransformation

10

2 Comments

I just finished building a data validation and cleaning framework for 170+ million FEC campaign finance records. Quick question for data engineers: 1. When does “cleaning data” stop being a script and start being a system? Mine wasn’t a modeling problem. It was a rigor and trust problem at scale. Instead of writing one massive cleaning script, I designed a 9-stage ETL pipeline with validation checkpoints at every step. If any stage failed validation, the pipeline stopped. No silent fixes. No downstream surprises. Here’s how the pipeline actually worked: I started by validating the raw FEC files themselves, checking schema integrity, headers, and row counts so I knew I was starting from a reliable baseline. From there, I narrowed the dataset to presidential-only transactions by identifying the relevant candidates and committees and removing tens of millions of unrelated records. Once scoped correctly, I validated critical fields like transaction amounts, dates, and committee IDs, removing records that failed basic integrity rules. I then tackled duplication, combining exact-match logic with fuzzy matching to catch near-duplicates while preserving legitimate repeat donations. Geographic data came next: standardizing states, territories, and military addresses so every contribution mapped cleanly to a valid geographic unit. I then segmented records by election cycle and quarter to ensure the temporal patterns aligned with known election behavior. The most subjective work followed. I normalized messy occupation and employer fields using fuzzy matching and external references, consolidating thousands of variations into consistent categories. Finally, I enriched the data by linking every record to candidate and committee master data, and stopped the pipeline if linkage rates fell below acceptable thresholds. Throughout the process, every stage had a hard validation gate before data was allowed downstream. The outcome: • 98.9M clean, analysis-ready records • 99.99% record-level validation pass rate • 99.84% alignment with official FEC totals • Zero unexplained row loss • Fully reproducible pipeline This dataset now supports analysis of $22B+ in campaign contributions, but the real win wasn’t the dashboards, it was delivering data stakeholders can trust without second-guessing the pipeline. So I’m curious: 2. Do you stop pipelines when validation fails, or log and move on? Data engineering isn’t about how much data you move. It’s about how much confidence you deliver. #DataEngineering #ETL #DataQuality #DataValidation #Python #PostgreSQL

29

1 Comment

Lately, I’ve started thinking about AI less as a replacement for analytics work and more as a force multiplier within my workflow. Recently, I was helping support a business request involving several downstream views and a reference table being migrated into Snowflake through a Coalesce pipeline. On the surface, the request sounded straightforward. Build the pipelines, populate the tables, validate the outputs, and make the data usable for the business user. But once I got into the work, the hidden complexity started showing up. Validation failures began appearing in staging tables. Then downstream integration layers started failing. Then MART objects weren’t materializing correctly. Some of the issues traced back to surrogate key behavior, null values, downstream dependencies, and naming propagation across STG → INT → MART layers. At different points, I had to manually hardcode SQL, implement MD5-based logic to stabilize object creation, validate lineage assumptions, QA downstream outputs in Snowflake, and work through deployment limitations while still ensuring the final structures actually worked for the end user. Throughout the process, AI was sitting beside me almost the entire time. Not replacing decisions. Not replacing validation. Not replacing business understanding. But accelerating the workflow. I used it to iterate through SQL approaches faster, troubleshoot errors quicker, test assumptions, validate logic, and think through possible solutions while moving from one blocker to the next. The interesting part is that the final outcome still came down to human judgment. The tables either populated correctly or they didn’t. The downstream dependencies either worked or they didn’t. The business user either received usable data or they didn’t. That’s why I’ve started viewing AI as a workflow multiplier in data analytics and analytics engineering. Not because it removes the need for expertise, but because it amplifies the speed and execution capability of people who still have to understand the business problem, the environment, and the downstream impact of the work. For me, that’s where the real value has been: moving faster from raw data problems to business-ready solutions.

5

🧠 AI Buzzword Wars: What's Real, What's Hype? [ 📊 Check out the dashboard: https://lnkd.in/gawXpRVv] [🔗 Read the full story in my medium article: https://lnkd.in/gzpfAMZT] I recently pulled together a Tableau dashboard comparing Google search trends and GitHub activity for today’s hottest AI buzzwords like LLM, RAG, AutoML, MLOps, and Vector databases. 🔍 Google Trends shows what people are curious about. 👩‍💻 GitHub data shows what engineers are building. Some surprises: ▪️ LLM dominates GitHub but trails AutoML in Google interest until recently. ▪️ RAG and Vector DBs exploded post-2023 : hype or real utility? ▪️ AutoML has surprisingly steady Google traction since 2020. This was a fun exercise in using publicly available data to visualize how AI conversations evolve and what actually translates into code. DM me if you would like the python script I used to webscrape the data from GitHub and Google search trends! :) Would love to hear : What other buzzwords would you add? #DataViz #AI #Tableau #MLOps #LLM #RAG #DataScience #GitHub #Trends #DataEngineering #OpenSource #GoogleTrends #Analytics

10

We are often drowning in data but starving for unique signals. Google Research just released GIST, a new framework for smart sampling. The core challenge in data engineering is selecting a subset of data that is both useful and diverse. Usually, these two goals conflict. If you pick the most useful points, you often end up with a cluster of redundant information. GIST provides a provable way to balance this. It identifies high-utility data points and establishes a conflict radius around them, any redundant data within that radius is filtered out to ensure maximum information density. This ensures every row in your training set or every document in your RAG prompt adds unique value. Practical takeaways for data teams: 1) 𝐌𝐞𝐚𝐬𝐮𝐫𝐞 𝐫𝐞𝐝𝐮𝐧𝐝𝐚𝐧𝐜𝐲, 𝐧𝐨𝐭 𝐣𝐮𝐬𝐭 𝐯𝐨𝐥𝐮𝐦𝐞. A dataset that is 50 percent smaller but 100 percent unique is often more valuable for model training. 2) 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞 𝐲𝐨𝐮𝐫 𝐜𝐨𝐧𝐭𝐞𝐱𝐭 𝐰𝐢𝐧𝐝𝐨𝐰𝐬. Use GIST-style logic to prune similar search results before feeding them to an LLM to save on token costs. 3) 𝐒𝐭𝐨𝐩 𝐦𝐚𝐧𝐮𝐚𝐥 𝐟𝐢𝐥𝐭𝐞𝐫𝐢𝐧𝐠. Moving toward algorithmic sampling reduces human bias and ensures your data covers the entire landscape of edge cases. I would apply this to our vector database ingestion. Instead of indexing every single document, I would run a pruning pass to eliminate semantically redundant entries. This keeps our RAG retrieval clean and reduces inference costs. #DataEngineering #MachineLearning #GoogleResearch #RAG #BigData Source: Google Research Blog, January 23, 2026, https://lnkd.in/gZ-h-4gH

18

The power of MSFT teams and genie, truly, genie is right now the best NLSQL agent that can dig through your data inside databricks to get you an accurate answer to your query. There is very minimum development and maintenance associated with creating and running it and it's damn effective. I personally am using genie to monitor my db system tables to make sure my team doesn't waste compute on useless scheduled ETL runs.

8

Excited to share that my Temporal webinar — "From Chat to Docs with Agentic AI" — is now available! The problem we were solving: support channels are full of tribal knowledge — corrections, workarounds, code snippets — but none of it makes it back into the docs. What that means is your documentation slowly drifts from reality. So we built agentic workflows with Temporal and LLMs that close that loop automatically. The system monitors chat threads, extracts structured insights, and opens PRs to update documentation — no manual curation needed. I cover the architecture, the AI extraction pipeline, and how Temporal's durable execution keeps these long-running processes reliable. Big shoutout to Shubham Shukla for being an amazing partner on this work, and to Julie Creamer for the support and leadership that made this possible at Nordstrom. Thanks to Ethan Ruhe and Hannah Short at Temporal Technologies for the opportunity and for helping put this together. Also — I'll be at REPLAY May 5–7 at Moscone South, San Francisco. Excited to learn from other great teams in the Temporal community. See you there! YouTube: https://lnkd.in/ga73Y45A Webinar page: https://lnkd.in/gRTzwMC4 #AI #AgenticAI #Temporal #DurableExecution #LLM #SoftwareEngineering

32

2 Comments

I just hit my one-year mark as a manager 🎉! It’s been a busy, challenging, and incredibly rewarding year. In this article, I reflect on three key themes that have shaped my experience leading a data team: 🔹 Prioritization – aligning the team on what truly matters 🔹 Empowerment – enabling growth and unblocking with context 🔹 Recognition – celebrating wins, big and small If you’re navigating your own transition into data leadership (or thinking about it), I hope this reflection is helpful. I’d also love to hear your perspective — what lessons stood out to you in your first year as a manager? Let me know in the comments below. Thank you, Towards Data Science, for posting my new article!

73

11 Comments

As a BI analyst, I've always worked with data that's already clean and ready. But how does it get there? That curiosity led me to build my first data engineering project, and wow... I learned SO much. What I Built: A stock data pipeline that processes both real-time and historical data. It's not perfect, but it works, and more importantly - I understand how it works now! The pipeline: → Pulls live stock quotes through Kafka → Loads historical data in batches with Airflow → Transforms everything with DBT → Serves data to Power BI dashboards (finally, my comfort zone 😅) My Learning Journey: Coming from BI/analytics, I had to learn: - How data actually flows before it reaches my dashboards - Why sometimes "real-time" isn't always necessary (batch is often smarter) - Setting up infrastructure with Docker (lots of trial and error) - Writing Python beyond pandas (Kafka producers/consumers were intimidating!) - Understanding the difference between data engineering and analytics Honest Moments: ❌ Kafka confused me for days (still learning tbh) ❌ Made rookie mistakes with data types and timezones ❌ Spent hours debugging Docker containers ✅ But each error taught me something valuable What Surprised Me: You don't need to be a "senior engineer" to build data pipelines. Tools like Docker and Airflow abstract away so much complexity. If you understand SQL and basic Python, you can start learning data engineering. Why I'm Sharing This: If you're a BI analyst, data analyst, or anyone curious about the "upstream" side of data - this project showed me it's totally approachable. You already understand the data modeling part (Silver/Gold layers felt natural!), it's just learning the ingestion and orchestration pieces. I documented everything - including my mistakes and "aha!" moments - in this Medium article: 💻 GitHub : https://lnkd.in/ggFhWMF3 📝 Read my learning journey: https://lnkd.in/gWydqG43 For other BI/analytics folks exploring data engineering: What's holding you back? Happy to share what helped me get started! #DataEngineering #BIAnalyst #CareerTransition #LearningInPublic #DataAnalytics #ModernDataStack #TechLearning #DataScience #Kafka #Airflow #DBT

25

𝐀 𝐭𝐨𝐨𝐥 𝐭𝐡𝐚𝐭 𝐠𝐞𝐧𝐮𝐢𝐧𝐞𝐥𝐲 𝐜𝐡𝐚𝐧𝐠𝐞𝐝 𝐡𝐨𝐰 𝐈 𝐰𝐨𝐫𝐤. As a TA for Data Engineering, I juggle Snowflake, pipelines, student reviews, and demo prep. Snowflake Cortex Code has become my go-to for all of it. 𝐖𝐡𝐚𝐭 𝐦𝐚𝐤𝐞𝐬 𝐢𝐭 𝐬𝐭𝐚𝐧𝐝 𝐨𝐮𝐭: 𝐏𝐫𝐞𝐛𝐮𝐢𝐥𝐭 𝐒𝐤𝐢𝐥𝐥𝐬 : Domain-specific skills for data governance, dynamic tables, cost intelligence, data quality, Streamlit, Iceberg, semantic views, dbt, and more. Like having a Snowflake consultant on call. 𝐒𝐮𝐛-𝐀𝐠𝐞𝐧𝐭𝐬 & 𝐁𝐚𝐜𝐤𝐠𝐫𝐨𝐮𝐧𝐝 𝐀𝐠𝐞𝐧𝐭𝐬 : Specialized agents work in parallel. one exploring code, another planning, another validating SQL. Background agents run while you keep working. 𝐌𝐨𝐝𝐞𝐥 𝐅𝐥𝐞𝐱𝐢𝐛𝐢𝐥𝐢𝐭𝐲 : Multiple underlying models. Pick the right one for deep reasoning or quick generation. 𝐖𝐡𝐚𝐭 𝐬𝐞𝐭𝐬 𝐢𝐭 𝐚𝐩𝐚𝐫𝐭 𝐟𝐫𝐨𝐦 𝐂𝐥𝐚𝐮𝐝𝐞 𝐂𝐨𝐝𝐞? Cortex Code is Claude Code built natively into Snowflake with prebuilt skills, Snowflake-aware agents, and direct warehouse access. It understands your data, warehouses, pipelines, and costs. No general-purpose tool matches that. 𝐈𝐧 𝐩𝐫𝐚𝐜𝐭𝐢𝐜𝐞:  • Planned an entire multi-tenant streaming platform in one session : architecture, governance, testing, CI/CD, all with actionable checklists  • Cost analysis, data profiling, and lineage tracing without leaving the terminal  • Streamlit dashboards for class demos using the dedicated skill  • Explored dbt project lineage, validated models, and debugged pipelines directly through the dbt skill This lets me focus on teaching, not boilerplate. Massive appreciation to the Snowflake team. This is what developer tooling should feel like thoughtful, powerful, and deeply integrated. #DataEngineering #Snowflake #CortexCode #BuildInPublic #AI

20

1 Comment

【近況報告】 (English follows Japanese.) Tableau DATA SABAR に興味を持ち、受��を始めました。😏 　WEEK0から、WEEK8まで全9週間で構成されており、Tableau の知識とスキルを底上げするプログラムになっています。🤔 　詳しい説明は、公式サイトに譲りますが、WEEKはDATA SABAR とは何か？に始まり、認定要件を中心とした、プログラムの全体像の解説でした。 　このプログラムの最大の特徴は、師匠選びです。 　師匠選びにあたり、こちらの記事を読み、そもそも、何故、DATA SABAR認定を得たいと思っているのか？どんなDATA SABAR を目指すのか？を明確にする必要があると感じました。 　受講を進めながら、これらを明確にしていこうと考えています。🤔 DATA SABAR https://datasaber.world/ DATA Saber 説明会でお話していること　ー師匠探しー https://lnkd.in/g5nNKCyV [Updates] I've started taking the Tableau DATA SABAR programme. 😏 　It's a 9-week programme, from WEEK to WEEK8, that aims to build solid foundations in Tableau. 🤔 　I'll leave the details on the official website. week0 discusses what DATA SABAR is and overview of the programme. 　One of the key features is selecting your master, who eventually supervises the last test.  　Upon selecting a mentor, I read the blog post and felt the need to clarify my motivations to become a DATA SABAR and what kind of DATA SABAR I aspire to be in the first place. 　I intend to work on these as I proceed to learn. 🤔 DATA SABAR https://datasaber.world/ DATA Saber briefing session -How to find your mentor- https://lnkd.in/g5nNKCyV

1

Can we extract new insights from a dataset originally designed for another purpose? 🛒 Last week, I came across a Instacart dataset from Kaggle. At first glance, it looked like a typical preprocessed dataset for product-level analysis—aisle trends, reorder rates, etc. But then I asked myself: Can we dig deeper and extract meaningful user-level insights? That question led me to build a full analytics project focused on understanding how users shop, return, and churn. 🔍 Instacart Funnel & Retention Analysis Project Here’s what I did: ✅ User Funnel Simulation Used order_number as a proxy for user lifecycle to map stages like first-time, repeat, and loyal. ✅ Behavior-Based Segmentation Built K-Means clusters using first-order features like product diversity, order size, and shopping time. → Identified 5 distinct personas from “Late-Night Browsers” to “High-Intent Bulk Shoppers.” ✅ Churn & Retention Modeling Applied Kaplan-Meier and Cox models to estimate retention across clusters—revealing that first-order behavior can predict long-term engagement. ✅ Live Looker Dashboard Created an interactive dashboard in Looker connected to BigQuery. Users can filter by segment, churn status, and order count. 📁 Full project & code: https://lnkd.in/da-SVgHX 💡 Project Highlights Transferred & processed 713MB+ of data via API: Kaggle → GCS → BigQuery (cutting 80% of manual load) Modeled user retention using survival analysis methods from retail Unlocked user insights from a dataset originally built for product trends

18

2 Comments