Kevin Densmore

I think people really underestimate what data professionals do. Like an (apache)iceberg, people really only see the analysis or dashboard - what’s above the water. There have been many times where I’ve tried to hand-off data-centric processes to folks, but I’d end up jumping back in given the complexity of debugging, investigating, and generally maintaining fragile systems. Thank your data analyst/engineer/scientist today - they’re busting ass to keep things afloat.

7

Most AI initiatives don’t fail because of weak models. They fail because intellgence is added at the wrong layer. Bigger prompt windows and more dashboards don’t create smarter systems — they create noise. Context isn’t a transcript. It’s a compiled, decision-ready view. That’s where SCIKIQ’s bolt-on intelligence fits in: Sits on top of your existing data platforms Adds a conversational and agentic intelligence layer Retrieves only what’s needed, when it’s needed Turns data into instant, reusable data products No rip-and-replace. No prompt stuffing. Just intelligence that scales. AI doesn’t get better by remembering everything. It gets better by remembering what matters. #AgenticAI #EnterpriseAI #DataPlatforms #AIArchitecture #ContextEngineering #SCIKIQ #GenAI #DecisionIntelligence

43

1 Comment

[NEW SERIES]: Putting Agents in your Data Architecture - Are we Ready? 🦞 I have been spending time reading research papers, blogs & understanding various system internals to address a question. "Are our existing data platforms ready to support Agentic workflows?" Traditional data platforms were built for kinda deterministic pipelines. - Engineers design data pipelines - Systems execute scheduled jobs (Apache Spark, Flink) - Queries tend to be bounded by human speed and reasoning The focus has been on repeatability and importantly stability. However, agentic workflows introduce a very different pattern in terms of how they will work inside our data architecture. Agents don’t execute a fixed pipeline. They explore systems dynamically. - They probe schemas - Issue iterative queries - Test multiple hypotheses - Refine strategies based on intermediate results This creates an entirely new class of workload for data platforms. So, we need to think of architectural primitives that can support these capabilities. The first primitive I want to talk about in this series is: ISOLATION! If agents are allowed to experiment with data (whether rewriting queries, testing transformations, evolving schemas), they need isolated environments. Agents may evaluate many “what-if” scenarios in parallel, each representing a speculative path toward solving a task. For that, they need safe environments to experiment and fail without consequences. So, How do we provide isolation for agents exploring large data systems like #lakehouse? ✅ Branch-able data states: allow agents to fork isolated copies of datasets to test hypotheses without touching production ✅ Snapshot-based versioning: enable agents to explore historical or point-in-time views of data safely ✅ Fast rollback mechanisms: since most speculative agent paths fail, systems must discard branches instantly ✅ Copy-on-write storage: create thousands of lightweight isolated environments without duplicating entire datasets. In many ways, the lakehouse with formats like Apache Iceberg might already be closer to agent-ready data infrastructure than we realize. And we already see several data systems already moving in this direction. For e.g. Neon has branching databases, allowing developers (and increasingly AI agents) to spin up isolated copies of production data instantly. Replit's snapshot engine allows AI agents to create instant filesystem forks, experiment in isolated environments, and roll back changes safely. Bauplan uses copy-on-write branching and commit semantics, allowing agents to run pipelines on temporary branches and merge changes only after verification. I am working on a new blog series that will go deep into these primitives. Helpful references in comments. #dataengineering #softwareengineering #ai

145

17 Comments

Is your AI Usage moving faster than your data Governance? The dbt Labs "State of Analytics Engineering 2026" report just dropped, and the Report reveals that 71% of users cite hallucinated outputs reaching stakeholders as a top concern. Here are the 3 biggest shifts every data leader and practitioner needs to keep in mind: 1️⃣ Acceleration vs. Trust: 72% of teams now use AI-assisted coding daily. While we’re shipping code faster than ever, the "Trust Gap" is widening. 83% of organizations now prioritize data trust above all else 2️⃣ Infrastructure is Outpacing Budgets: Warehouse and compute costs are rising for 57% of companies, but team budgets aren’t keeping pace. We are being asked to do more with less, leading to a focus on "Speed vs. Trust" rather than just "Speed vs. Cost." 3️⃣ The "Context" Crisis: AI can write SQL in seconds, but it can’t invent business logic. 2026 is the year of Context. Without robust metadata, documentation, and clear ownership, AI-generated insights are just "fast hallucinations." The Bottom Line: AI won’t fix a messy foundation; it only makes the lack of discipline more visible. This year, the winners won't be the ones who code the fastest, it'll be the ones who build the most reliable systems. What’s the biggest challenge your team is facing in 2026? Are you feeling the "Trust Gap"? Let’s discuss below! 👇 🚀 Find more articles like this everyday at https://lnkd.in/esvV4ShW #DataEngineering #AnalyticsEngineering #dbt #AI #DataStrategy #ModernDataStack

4

1 Comment

🔥 [New Blog] "Unstructured To Structured : Using Fabric AI Functions For Contextual Data Quality Check" I read an excellent blog a while ago on how LLM can help with Data Quality checks. We typically use completeness, consistency, accuracy, timeliness etc to check DQ. But what about contextual accuracy of the record? This is where LLMs can really help. In this blog, I use #AIFunction in #MicrosoftFabric to check contextual DQ on real world data. This is the first blog on this topic. I will explore it more in the next blog with eval harness, operationalization etc. 🔗Blog : https://lnkd.in/gzVpXtJ5 📷Image credit : https://rdcu.be/eyVnU 🔗Blog that inspired this : https://lnkd.in/g-Mb97aZ

131

9 Comments

Thinking about Data Governance in K-12 often conjures up images of massive, dusty policy binders that no one actually reads. But in reality, effective governance isn't about bureaucracy, it’s about trust. When your SIS, LMS, and HR systems all define "enrollment" differently, leadership loses confidence in the dashboard. The secret? Start small and stay consistent. The 15-Minute Governance Ritual Instead of grueling half-day committees, aim for a 15-minute monthly stand-up to: - Review requested definition changes (e.g., new state rules for chronic absenteeism). - Consolidate duplicate reports. - Record decisions in a centralized Data Handbook. 3 Steps to Start Today: - Pick 8–10 High-Impact Terms: Define "Attendance" or "On-Track" in plain language. - Assign Owners: Every definition needs one primary human responsible for its integrity. - Create Guardrails: Transition from manual spreadsheets to automated validation rules. Data governance shouldn't be a hurdle. It should be the "quiet engine" that makes your district's data reliable and actionable. https://lnkd.in/gfqvMiaB

9

What does it take to build a unifying strategy for scale? See 2025 brochure here: See 2025 agenda here: https://lnkd.in/e7ZJ_4b2 At FIMA, Suraj Rajan, Field CTO, Financial Services EMEA at Snowflake, will explore how aligning data, governance, and architecture enables AI adoption, empowering intelligent decision-making and driving commercial outcomes in the keynote Unifying Strategy for Scale. 

24

The ElevenLabs story in the Financial Times highlights something I’m seeing repeatedly across UK AI scaleups. The challenge isn’t innovation anymore. It’s execution at speed. Once usage takes off, teams suddenly need to: Scale ML & data infrastructure Control inference and cloud costs Add reliability, observability, and governance Deliver now, not after a 6-month hiring cycle This is where many strong AI businesses stall — not from lack of vision, but lack of delivery bandwidth. The most effective teams I’m seeing are: Deploying specialist Data & AI contractors into short, outcome-driven squads Using contract talent to absorb spikes in demand post-funding Keeping permanent hiring focused on long-term core roles Speed wins this market. But only if it’s controlled. If you’re scaling AI or data platforms right now, I’m curious: What’s been harder hiring fast enough, or delivering safely at pace?

3

2 Comments

Example of an effective use of big data. Analysis of 1.1 million podcast episodes, on the substance, context and framework. From this, we an infer generalizability, shared narrative, etc. Dig in and enjoy. Imagine the same context in MSK big data research, Chuck Thigpen Trevor Lentz Chad Cook Steven Kamper • 🎙️ Massive new dataset: Introduces SPORC, a 1.1M‑episode corpus capturing nearly all English‑language podcasts released via public RSS feeds in May–June 2020. • 🧠 Rich multimodal data: Includes transcripts, metadata, inferred speaker roles, and prosodic audio features; 370K episodes also include diarized speaker turns. • 🤖 High‑quality transcription: Whisper-based ASR achieves <10% word error rate; filtering removes repetition artifacts to maintain transcript quality. • 🗣️ Speaker insights: Combines diarization + NLP to identify hosts and guests, enabling analysis of conversational structure and cross‑show guest networks. • 🧵 Topical landscape: Topic modeling reveals coherent clusters (e.g., sports, religion, business) and cross‑category communities (e.g., spirituality, self‑improvement, COVID‑19, racial justice). • 🌐 Guest‑based social network: BUSINESS, SPORTS, and NEWS form dense, insular guest‑sharing networks; RELIGION and SOCIETY are far less interconnected. • 🔗 Community structure: High modularity in SPORTS and BUSINESS indicates strong within‑category guest ecosystems; others are more diffuse. • 📰 Responsiveness to events: Podcasts vary widely in how quickly and intensely they respond to major news events (e.g., George Floyd’s murder), revealing differences in attention, audience, and mission. • 🧩 Research potential: SPORC enables new work on information diffusion, community identity, misinformation, and long‑form audio communication at unprecedented scale.

11

3 Comments

🎙️ Airing tomorrow: a new ER/Studio podcast with Nicola Askham (The Data Governance Coach) and Jamie Knowles (ER/Studio Product Director). The big question: “JSON: As a Data Governance Person, should I care?” They’ll explain what JSON is, where it shows up in modern data platforms, and why it matters for governance leaders who want to connect business accountability with technical reality. 👉 Don’t miss this one → dropping Oct 14. And if you haven't, ⏯️ check out some of our latest content with Nicola in the Data Product Delivery Series. https://lnkd.in/eBs7u7n6 #ERStudio #JSON #DataGovernance #Podcast

18

2 Comments

There are so many ongoing conversations about AI in the data science and analytics world right now that there could easily be a track about each of these sub-themes. This track will be getting into all sorts of nitty-gritty goodness about to build and evaluate trusted systems, how to give them the context they need to be effective, and even how to rethink how our teams operate. You won't want to miss it!

41

2 Comments

Turn fragmented data into a trusted business asset with a clear governance blueprint. Define roles, enforce policies, and track quality metrics so your teams spend time on insight — not rework. Aidentity.uk brings architecture expertise and practical data solutions to speed analytics and AI adoption. Learn how to operationalise governance for reliable, reusable data: https://wix.to/ck1zLkY 🔍📊 #DataGovernance #DataStrategy #AIReady

1

What is a Next Gen Semantic Layer and why is it important? A Next Gen Semantic layer unfies the worlds of data management, information management, and knowledge management. It's a platform to allow for the concepts of 'meaning' and 'context' to be widely shared across an organization. In doing so, it will allow for organizations to leverage the insights buried in their legacy structured databases by GenAI based agents. It will also allow for companies to apply structure to all of the insights buried in their highly unstructured data sources, and to govern this data at scale. It will function as the connective tissue between the worlds of GenAI and BI, and will be the arbiter of 'truth' across all business domains. Powered by AI, it will act as a universal translation layer that will enable accurate and consistent cross-domain insights. It will sit on 'top' of physical data stores and support both centralized and decentralized patterns of data management - depending on the use case. It will serve as both a rules engine, and eventually, a recommendation engine, to data fabric architectures where all data integration and access workloads are unified under a single (and heavily virtualized) operational plane. It will support data stewardship workflows to allow for 'human in the loop', where necessary, but it will also heavily rely on AI-based solutions to automate many data management tasks, particularly data classification and profiling. It will be a critical component to enable the 'activation' of metadata, at scale, within an organization. It will slowly begin to unify several disparate disciplines of data management into a single workflow, where the functional distinctions between MDM, data catalogs, and data quality tools are slowly eroded. But, this is just my opinion. What do you think? Jessica Talisman Juan Sequeda Ole Olesen-Bagneux Piethein Strengholt Samir Sharma Jon Cooke Jean-Georges Perrin Lulit T. Paco N. John O'Gorman Sanjeev Mohan Andreas Blumauer Ramona C. Truta, MSc

355

94 Comments

"What Is Retrieval-Augmented Forecasting (RAF)? On a very high-level view, instead only leaning on what a model learned in training, RAF lets the model actively look up concrete past situations similar to the current one and use their outcomes to guide its prediction." https://lnkd.in/gH4uQbJJ

3

🔍 RAG Isn’t Just a Buzzword—It’s the Future of Scalable, Accurate AI In the ever-evolving landscape of AI, RAG (Retrieval-Augmented Generation) is emerging not just as a trend—but as a paradigm shift in how we approach large language models. While foundation models like GPT-4 or LLaMA are incredibly powerful, they’re limited by the static nature of their training data. That’s where RAG comes in—bridging the gap between pretrained knowledge and real-time, dynamic information. 💡 What Is RAG, Really? At its core, RAG injects fresh, relevant context into AI prompts by retrieving up-to-date external data (from databases, documents, or APIs) before generation occurs. Think of it as combining the power of search with the fluency of generation—on demand. ✅ Why RAG Matters Here’s why RAG is more than a technical trick—it’s a game-changer: - Higher Accuracy, Less Hallucination: By grounding responses in real-world documents, RAG reduces the risk of fabricated answers. - Always Up-to-Date: No need to retrain the core model—RAG can surface the latest data in real time. - Cost-Effective at Scale: Avoid expensive fine-tuning while improving quality—ideal for legal, customer support, finance, and more. Ecosystem Ready - Works with modern AI tooling: GPT-4, LLaMA, Pinecone, FAISS, LangChain, Weaviate, and beyond. ⚠️ But It's Not Plug-and-Play RAG isn’t without its challenges: - Context Window Limits: You can’t cram unlimited data into a prompt. - Latency: Real-time retrieval adds milliseconds—sometimes too many. - Retrieval Quality: Garbage in, garbage out. Poor indexing = poor results. 🧠 Final Thought RAG represents a critical evolution in AI architecture—one that prioritizes relevance, flexibility, and trust. It’s not just the future of GenAI—it’s the path toward usable intelligence. Are you already exploring RAG in your stacks? What tools or use cases have you found most effective? #AI #RAG #RetrievalAugmentedGeneration #GenerativeAI #LangChain #VectorSearch #GPT4 #AIInfrastructure #AIEngineering #KnowledgeManagement #OpenSourceAI #FutureOfWork #AIProductDevelopment

242

39 Comments

A must-read for anyone in Data Governance or Management. Tony shares so many great insights and some inconvenient truths about our field. Thought-provoking and highly relevant!

5

2 Comments

The UK public sector is stuck in a data mess. Not because of a tech shortage — but because it still runs on scripts. The future? Autonomous data agents. They don’t just automate tasks. They achieve outcomes. → Clean data → Faster insights → Less manual work Engineers stop coding. They start commanding. This is the agentic shift. #AgenticAI #AIagents #PublicSectorInnovation #GovTech #DataOps #DigitalTransformation #AutonomousAI #AIinGovernment #AIforGood #SmartData #RplusAnalytics

15

2 Comments