Romit Jadhwani

A pattern I keep seeing in platform decisions: Teams optimize for *technical capability* before asking about *decision frequency*. Streaming pipelines. Always-on compute. Sub-second freshness. But the actual decisions happen: • once a day • once a week • sometimes once a month We learned this lesson painfully during migrations: If no one acts on data in minutes, real-time becomes an expensive habit. Start with how fast decisions are made. Earn your way to complexity. What’s one “real-time” system you’re running that nobody checks in real time?

13

Snowflake Cortex Search is redefining how we interact with data — blending AI, vectors, and search natively within the warehouse. It’s powerful, fast, and deeply integrated. And here’s the real magic: what if you didn’t even have to think about how to set it up? That’s where #Frosty (PATENT PENDING) comes in. At Gyrus, we’ve built automation that makes #CortexSearch effortless. #Frosty automatically discovers the right data to enable, determines which columns should be used for vector indexing and which for filtering, and even adapts to your data’s freshness — configuring the service to refresh results exactly when they should be. With #Frosty, #CortexSearch no longer waits on enterprises — it’s automatically ready wherever it fits best. #Snowflake #CortexSearch #Frosty #AI #Automation #AIAgents #AutoPilotForSnowflake

8

Most conversations about end-to-end AI and model training still focus on scale. But scale alone doesn’t get you to production AI. What actually matters across the full model training lifecycle is signal: high-quality labeled data, clear data provenance, human-in-the-loop evaluation, and retraining pipelines supported by strong MLOps, deployment, and monitoring. At Shutterstock, we offer solutions to AI teams across the entire training lifecycle from licensed training data, custom training datasets, and multimodal data curation to model training, fine-tuning, model evaluation, benchmarking, and continuous model improvement. We operate as an AI lifecycle partner, not a point solution, supporting enterprise AI teams with data sourcing and creation, refinement, human preference signals, and qualitative evaluation that models can reliably learn from. The outcome isn’t just better benchmarks. It’s production-ready and always improving models with stronger model alignment, reduced regressions, and greater confidence at deployment. Get in touch with our team to learn more: https://lnkd.in/guCmrqxm #ProductionAI #EnterpriseAI #ModelTraining #Shutterstock #FuelGreatWork

3

AI-Ready Data Is Not What You Have. If AI fails in your enterprise, the board will blame data. But if you deliver AI-ready data, you become the foundation of every successful AI initiative. That’s not a technical nuance - it’s a career-defining moment for data leaders. In my new Substack, I share what “AI-ready” really means and a practical checklist to get started: https://lnkd.in/eeVgu5y9

52

30 Comments

If you find yourself with some time on a Friday afternoon, give this tool a spin. Our team built it using a novel approach based on a paper we discovered, published last October in conjunction with Colgate-Palmolive. It creates persona-led product review panels that deliver instant landing page feedback. The tool has been maxed out since 𓃋 Tyler Gargula released it yesterday, so it's been really great to see the reaction. Full write up and tool linked below.

12

1 Comment

Attending Snowflake BUILD 2025 was such an incredible experience. It’s amazing to see how Snowflake is re-imagining the future of AI and data. Here were my top 5 takeaways from the conference: 1️⃣Building the agentic future: AI agents moving from concept to production. 2️⃣Cortex AISQL: natural language queries, no prompt engineering needed. 3️⃣ Cortex Agents: from queries to multi-step reasoning and automation. 4️⃣ Managed MCP Server: bridging AI tools through standardized connectivity. 5️⃣ Gen2 Warehouses: a data foundation built for real-time, enterprise-scale AI. Watch the full breakdown in my video 🎥 And if you’d like to learn more about Snowflake Intelligence, you can check out the link here: https://lnkd.in/gnRkJPv2 #SnowflakePartner #SnowflakeBUILD

303

31 Comments

🚀 Big Shift in Enterprise AI 🤝 Palantir Technologies x Anthropic — Claude is now available inside AIP. 🧠 Frontier LLMs can run securely within ontology-driven workflows. ⚙️ For Palantir Data Engineers, this makes work easier: faster pipeline debugging, smarter data exploration, automated documentation, and quicker insight generation. 📊 Complex SQL/PySpark logic, lineage tracing, and transform explanations become AI-assisted. 🔒 It solves the governance gap — powerful AI with auditability & compliance. ⚡ But it’s also challenging: prompt design, grounding models correctly, avoiding hallucinations, and architecting AI-driven workflows responsibly. 🎯 The path forward? Master Ontology + AIP orchestration + model behavior understanding. 🔥 The engineer who blends data engineering with AI reasoning will lead the next wave.

7

1 Comment

The data catalog, policy enforcement, and the semantic context AI agents need to reason about a business are collapsing into the same layer. Three signals from a Snowflake Executive Briefing Center conversation this month make the pattern hard to ignore. Collibra's deepening integration with Snowflake Horizon and its seat at the new Open Semantic Interchange consortium are quietly making governed AI the default state, not a separate program you bolt on later. Snowflake Semantic Views are the pathway connecting them. Three shifts worth naming: Policies authored in Collibra push down as native Snowflake row access and masking. Write the rule once, see it enforced everywhere it matters. Collibra Data Products publish into Snowflake Semantic Views and into OSI as governed YAML, two accelerators released this quarter. The same metric definitions your humans trust are the ones LLMs and agents read at runtime. Sensitive data detection extends into the unstructured documents fueling generative models, not just the rows and columns. The theme worth carrying: try it before you scale it. The whole pattern is testable on a Snowflake free trial in an afternoon. The planning conversation shifts after you do it. No longer "how do we govern AI?" but "what do we want our governed AI to do?" What is your first governed AI use case worth proving on the new semantic layer? Sources and a five step walkthrough in the first comment. #DataGovernance #AI #Snowflake

31

2 Comments

A perfect snapshot of the Model Context Protocol (MCP) landscape right now. 😂 There's a flurry of activity in building for MCP, but the end-user side of the equation looks pretty empty. Why? It's about two key market factors. The first is slow AI agent adoption: The primary use case for MCP is to allow different AI agents to communicate and collaborate. But mainstream adoption of sophisticated, autonomous agents by the general public and businesses is still in its early stages. We have the communication standard, but not enough agents using it in the wild yet. And then there are just too many developers building similar, competing agents within their own ecosystems. The groundwork being laid is crucial, but we won't see the users show up until AI agents become more integrated into our daily workflows and the builder community shifts from competition to collaboration. If these agent companies don't perform in the next year or two, some bubbles (smaller ones) are gonna pop. #AIAgents #MCP #ModelContextProtocol #AI

83

3 Comments

Seven West Media is accelerating audience growth and boosting ad revenue with the Databricks Data Intelligence Platform. By building a centralized AI Factory with Mosaic AI and Unity Catalog, Seven has achieved a 40% increase in daily active users on 7plus, generated $16M in incremental ad revenue through re-engagement campaigns, and halved the cost per click across marketing efforts. Real-time, AI-powered insights drive predictive analytics, smarter audience targeting, and self-service tools for teams company-wide—delivering market-leading results and setting a new standard in digital media.

1

If you're building AI agents for the enterprise, read this before you write a single line of code. Most agent conversations are still about prompts and models. I just went through “Building Effective Enterprise Agents” by BCG’s AI Platforms Group, and it is one of the most practical enterprise agent playbooks I have seen. Here are the biggest takeaways 👇 1. The problem is not intelligence. It is an engineering discipline. 75% of tech leaders fear “silent failure”. Spending on AI without P&L impact. The top blockers BCG sees in real deployments: - Siloed, unreliable enterprise data - Heavy governance and audit demands - Moving from PoC to stable operations - Integrating into legacy systems - Lack of serious evaluation setups 2️. Design for outcomes, not tasks Do not build an agent to “process invoices”. Build it to: “Reduce exception handling time by 30%.” BCG pushes outcome-first design: - Start with business KPIs - Break them into dependency trees - Identify where agents truly add value - Use simple automation where agents are overkill This avoids building “cool agents” that do not move the business. 3. Start simple. Most teams try to jump to “agent mesh” too early. BCG defines maturity horizons: - Constrained agents - Single agents - Deep agents (orchestrators) - Role-based agent teams - Agent mesh Enterprises today should focus on single agents and deep agents, tightly integrated with existing systems.  Complexity should be added only when simpler flows break. 4. You do not need “an agent”. You need an agent platform. BCG outlines 14 core components for production agents, including: - Data platform readiness - Context engineering - Short-term + long-term memory - Guardrails and safety layers - Model gateway for switching models 5. Evals are the real superpower You cannot improve what you do not measure. BCG shows how structured eval loops improved accuracy from ~50% to ~75% in an insurance case, translating into real revenue impact.  This is “hill climbing” with data, not vibes. 6. Buy, build, or hybrid? The answer is hybrid. - Buy for commodity, in-suite use cases - Build for differentiating, cross-system orchestration - Most enterprises will do both Where you build should follow data gravity, system gravity, and governance constraints. 7. Security model must change Enterprise agents are not just “apps”. BCG describes 5 interacting systems: - Experience (how users interact) - Environment (tools, APIs, data) - Policy (reasoning and control flow) - Runtime (infra) - Operations (monitoring, security, lifecycle) - New risks include prompt injection, tool misuse, hallucinations, and multi-agent deadlocks. Security and observability must be built in from day one. Big picture: 2025 was experimentation. The next phase is operational discipline, platform thinking, and outcome focus. Save this to read later and repost to share with your network. Follow Shubham Vora to learn more about implementing AI in your business.

98

69 Comments

AI & Market Research - Can AI take surveys for us? I haven't posted about too many uses of AI with Market Research to date, however I think this case study should get a mention. A recent study by PyMC Labs and Colgate‑Palmolive finds that large language models (LLMs) can closely simulate human responses in consumer product surveys—hitting ~90% of human test–retest reliability across 57 surveys (9.3k human responses). How it worked: Instead of asking a bot for a number, the team asked for a short written answer with a why. That text was then compared to five Likert “anchor” statements using embedding similarity (their Semantic Similarity Rating / SSR method). Result: 90% of human test-retest reliability. They mirrored distributions of responses across demographic segments such as age and income. Researchers emphasise that this doesn’t mean LLMs think like people, but their outputs can align with human behavioural patterns under defined conditions Want to try this? 1) Start with a low-risk, high-volume concept test you already run. 2) Write clear Likert anchors (1–5) as plain sentences. 3) Prompt an LLM for a short rationale per concept and then map that text to anchors using embedding similarity. 4) Create a small human holdout to benchmark; only proceed if your synthetic vs human gap is tight. 5) Be transparent: label synthetic vs human, document prompts, and keep humans in the loop for final calls. Imagine being able to test new tourism or retail products or ad concepts instantly across synthetic “audiences”, before spending on full panels. Huge potential for insight and speed — as long as we keep the human layer where it matters most: judgement, context, and creativity. This isn’t about replacing people; it’s more about "simulate → iterate → validate" so teams can test more ideas, faster, and spend human budget where judgement matters most. Full story below 👇 #AI #MarketResearch #Insights #DigitalTransformation #SyntheticData #Tourism #Innovation https://lnkd.in/eRBWtRcs

6

1 Comment

We’re learning a lot at Snowflake, very quickly, about what it actually takes to deliver high quality, high-performance agentic analytics in the real world. Rajhans Samdani and I collaborated to write this latest post, which captures some of those learnings, especially around the emerging “context layer” and why it is becoming foundational for trustworthy data agents. https://lnkd.in/g-8TMcmD What’s been most interesting is how consistently the same themes show up across customer engagements. Whether it’s Semantic Views, Cortex Analyst, our Agent Framework, or Snowflake Intelligence, the same pattern keeps reinforcing itself. You can’t get to reliable, scalable outcomes without a strong, well-defined layer of context that connects data, business meaning, and user intent. This is one of those moments where the pace of change is hard to overstate. The space is evolving quickly, and we’re seeing ideas move from early experimentation into real customer impact in a matter of weeks. Being able to take what we’re learning directly from customers, feed that back into the platform, and then see it improve results is what makes this work so exciting right now. If you’re thinking about how to build or scale agentic analytics, I hope you’ll find this perspective useful.

336

9 Comments

💡 When it comes to AI, quality isn’t optional—it’s everything. Working with a global technology leader recently reinforced something I’ve seen time and again: the success of AI models depends on the quality and integrity of the data behind them. Their challenge was familiar—sourcing qualified medical annotators, ensuring consistent labeling across multiple regions, and curating vast multimodal datasets while navigating cultural, linguistic, and regulatory nuances. What stood out for me in this engagement was how a domain-driven, human-centric approach transforms outcomes. By combining licensed medical experts, multilingual curators, and advanced annotation tools, we were able to: 💥 Achieve 95%+ first-pass acceptance through robust multi-tier quality control. ↘️ Reduce costs by 45%+ with offshore delivery and performance-based incentives. 📦 Deliver compliant, high-quality annotated datasets across the U.S., Japan, Korea, and India—ready for generative AI applications. 💫 The takeaway? AI performance isn’t just about models—it’s about the data feeding them. At Qualitest, we focus on QUALITY data at scale, empowering clients to build models they can trust, globally. #AI #DataAnnotation #GenerativeAI #QualityData #DataServices #HealthcareAI #MultimodalData #ThoughtLeadership

16

TL;DR: A/B tests validate. Counterfactuals scale. How are you thinking about this? As organizations move from traditional analytics to Agentic AI systems, one question keeps coming up: Should we rely on A/B testing or counterfactual inference to understand impact and guide optimization? Both matter - but they serve very different purposes. Evaluating impact isn’t just “did it work?” It’s: “what would have happened if the agent chose differently?” 🔹 A/B Testing Classic causal validation: controlled, randomized, clean comparisons. Works well for single, isolated decisions - but agentic systems operate in dynamic, multi-step environments with branching trajectories. A/B tests alone can’t capture this complexity. 🔹 Counterfactual Inference Estimates “what would have happened under alternative decisions” using tools like: • Causal forests • Uplift modeling • Synthetic controls • Off-policy evaluation These methods allow scalable, multi-step causal reasoning without needing dozens of live experiments. 🛒 Retail Examples • Personalized promotions: Estimate if the agent’s recommended offer increased incremental sales vs. what would have happened with a different incentive. • Store operations agents: Measure how much faster labor scheduling or replenishment tasks would have been if the agent suggested an alternative pathway - without needing dozens of live experiments. • Member engagement journeys: Understand the lift of an agent altering sequence, timing, or tone of outreach across touchpoints. ✅ Takeaway A/B tests anchor big decisions. Counterfactual inference scales intelligence across agentic AI systems. For complex real-world environments, counterfactuals are essential for causal reasoning at scale. #DataScience #CausalInference #AgenticAI #RetailAnalytics #MachineLearning Swati Kirti Chittaranjan Tripathy Yog Seetharama Fabiano Battaglin Subhasish Misra Felipe Morales Heerlein Jason Cho, PhD

69

12 Comments

I was chatting with Ranjit Nagesh about the pros and cons of Databricks' "Serverless" compute. We both really like what it means, but also believe that there are areas of improvement, such as the ability to limit the $ spent on a pipeline. Personally, I feel very comfortable with Serverless Jobs on "Standard" more, but I'd be a bit hesitant on using optimized without the cost controls. If on interactive, cost controls become even more important. I'd love to hear other people's feedback on what is working for them, and what is not working for them when it comes to Serverless. *Note to competing Databricks vendors: If you use this post to attack Databricks and/or promote your product, your comments will be likely deleted.

18

7 Comments

The 2025 MAD (ML, AI, and data) Landscape from Matt Turck and team is out, and it’s a major reset. They cut the logo count from 2,000+ to ~1,150 to make the map readable again, and expanded the footprint of companies driving the change: NVIDIA, Databricks, Snowflake, OpenAI, Anthropic. Structurally, this is a new map: - No standalone “open source” box - A real agent stack (platforms + infra) - Local / on-device AI as a first-class category - Cleaner flow from data → infra → ML/AI → agents/apps What stands out even more is consolidation. This year was the year of acquisitions, here are just some of them: - Featureform→Redis - dbt/Tobiko→Fivetran - Tecton/Neon/Tabular→Databricks - WhyLabs→Apple - W&B→CoreWeave - DAGWorks→Salesforce Curious what you’re spotting this year! Check it out: https://lnkd.in/eBhiEfnu

61

7 Comments

I think Databricks’ Genie Code is clearly less about introducing an entirely new product and more about formalizing the steady improvement of its AI development experience over the last year. Not only has the output quality improved over time, but features such as dashboard authoring, Lakeflow Spark Declarative Pipelines support, etc were under-the-radar additions that positively impacted development speeds, quality, and time-to-delivery. Some competitors have had to rethink or replace rough first-generation coding products, Databricks started with something usable and kept improving it, so hey, might as well celebrate with a new name too! Also, I believe the emphasis on more agentic capabilities points to where the platform is going next, especially as Databricks pushes toward a more unified AI product experience built around governed enterprise data.

200

11 Comments

Tuning Spark, Ray, or Dask for ML isn’t one-size-fits-all. In a new Domino Blueprint, Principal Solution Architect Sameer Wadkar shows how API-driven scaling complements autoscaling to right-size clusters on demand—cutting idle spend and speeding bursts for GPU/I/O-heavy work. Practical steps, a lightweight Python client, and Helm install included. If you care about FinOps and MLOps, this is for you. Read the blueprint: https://gag.gl/WwWso7

1