The Importance of Metadata in AI

Not enough people are paying attention to the significant shift in how organizations use metadata. The massive adoption of AI changes EVERYTHING. Before, metadata was this obscure data source that a few technical people in the weeds looked at. Now, metadata serves as the "highway signs" that AI references to know what "onramps and offramps" to utilize for its workflows in an organization's vast network of information. This will only grow as AI Agents (i.e., AI + tools) gain further adoption. Here are some signals in the broader market that make me believe this: 1. Data contract architecture is gaining adoption across enterprises as a mechanism to manage the changes to metadata between systems. 2. The rise of Apache Iceberg, whose primary value prop is how it manages metadata to speed up data lookups and management of data lakes. 3. Many data conference having talks around "ontologies" and "knowledge architecture" of metadata. 4. Conversations around Meta Grid, Data Mesh, and Data Fabric as means of mapping data assets (and their metadata) across tools and organizations. 5. AI moving away from just LLM chatbots to actually taking action with specific scopes and tools without a prompt. There is all this noise about AI this and AI that, but no discussion around the underlying infra that enables this work. Metadata is only going to grow in importance!

Metadata is essentially "data about data." It provides descriptive, structural, and contextual information, making other data easier to understand, locate, and use effectively. By capturing essential details—such as a dataset's origin, structure, purpose, relationships, and meaning—metadata enables data to be organised and contextualised. Currently, much of the focus in AI is on high-profile generative models, representing only the visible tip of the AI Iceberg. However, beneath the surface lies the foundation: data. What sits below determines what can happen above, so organisations must first organise their data by prioritising metadata. Metadata can be divided into four main types: 🔵 Descriptive Metadata: This includes information that helps identify and locate data. For example, a book’s metadata might contain the title, author, publication date, and genre. In a digital setting, descriptive metadata could include tags, keywords, or descriptions, making data easier to search and retrieve. 🔵 Structural Metadata: This describes how data is organised or formatted. For instance, in a database, it might define table relationships or document structures, helping to ensure data is correctly interpreted, stored, and processed. 🔵 Administrative Metadata: This encompasses information needed to manage data, such as data ownership, access permissions, or retention policies. Administrative metadata is crucial for data governance, ensuring data is properly maintained and protected. 🔵 Semantic Metadata: This connects data to meaning, especially in the context of AI and knowledge management. Using ontologies and knowledge graphs, semantic metadata establishes relationships and contexts, helping AI to "understand" distinctions in data—such as the difference between a "financial asset" and a "physical asset." Here is a key insight: Semantic metadata gives data meaning. By adding semantics to metadata itself, we make all metadata meaningful. While this might sound complex, it’s quite achievable in practice. By using ontologies and knowledge graphs, you can unify Descriptive, Structural, and Administrative metadata within a semantic framework. This creates a single Semantic Layer over all your organisation's data. AI can assist in building this Semantic Layer over your data, leveraging the general semantics of natural language. It can then use that Semantic Layer to interface more seamlessly with the specific semantics of your organisation's data when answering questions at runtime. The concept is simple, but implementing it requires time and effort—and time is running out. Organisations need to redirect resources from prototype AI projects and vanity showcases to the real task at hand: preparing their data to be effectively utilised by AI. ⭕ Getting Started: https://lnkd.in/eF8WGGGH ⭕ The AI Iceberg: https://lnkd.in/esNckcDV

AI needs a different kind of data management to succeed—letting go of neatly structuring the world and doubling down on metadata. For decades, corporate data management meant discipline: defined fields, taxonomies, and carefully crafted data models built to impose order. Information was captured, cleansed, and contained under the assumption that insight followed structure. That assumption no longer holds for today's AI systems (=LLMs and other generative models). They have little use for traditional tidiness. They train and infer not on tabulated records but on vast, unruly troves of unstructured content. What matters isn't order, but abundance, diversity, and context. Yet many organizations still treat AI as sophisticated BI or supervised ML. Investments flow into rigid structures and polished pipelines pursuing "AI-ready data." But AI isn't a more fancy dashboard. It serves a different purpose: learning patterns in highly unstructured data and dealing with ambiguity. Therefore, AI needs a different kind of data management that shifts from enforcing structure to enabling understanding. If unstructured data is AI's raw material, then metadata—the data about the data—is its essential scaffolding. In a world where AI trains on noise, metadata provides the signal. It identifies sources, flags permissions, captures provenance, encodes trust. It tells systems not just what content is, but who created it, in what context, and how credible it might be. It helps models distinguish satire from sincerity, guidance from opinion, sensitive from shareable. Data quality is fundamental as much as ever, but in a different way. Yes, AI is vulnerable to biases and factual errors. But fixing this hinges less on conformity to schemas and more on richness, representativeness, and reliability. Metadata becomes critical where AI meets legal, ethical, and regulatory demands: access controls, lineage, consent, auditability—these depend not on content structure, but on surrounding metadata, enabling responsible use of messy data. If unstructured content is the terrain, metadata is the map. The task isn't abandoning data management, but evolving it. Structured systems remain vital for transactions, but AI's promise lies in embracing the richness—and mess—of the real world, while building tools to navigate it wisely. Organizations that thrive in the AI era won't be those with the cleanest data warehouses, but those with sophisticated metadata ecosystems. This shift from data hygiene to data context represents not just a technical evolution, but a philosophical one—acknowledging that in a complex world, understanding often matters more than order.

Metadata is the backbone of data discovery, governance, and AI-driven insight. Yet, I see the same mistakes repeat across organisations: 1️⃣ Treating metadata as an afterthought Mistake: Capturing it only at the end of a project. Fix: Make metadata creation and enrichment part of the data life cycle from day one. 2️⃣ Inconsistent naming conventions Mistake: Mixing terms, abbreviations, and styles without standards. Fix: Establish and enforce a naming and tagging policy across the organisation. 3️⃣ Missing business context Mistake: Only storing technical details (columns, types) without explaining what the data means. Fix: Include business definitions, data owners, and usage examples in your metadata. 4️⃣ Ignoring relationships Mistake: Capturing isolated data assets without showing how they link together. Fix: Use taxonomies to show connections. 5️⃣ Letting metadata rot Mistake: Never updating it when data changes. Fix: Automate metadata harvesting and set governance checks to keep it fresh. 💡 Strong metadata doesn’t just make search easier, it fuels trust, speeds up projects, and unlocks AI capabilities.

One of the quietest but most important conversations in the Department of War is not about drones, LLMs, or autonomous agents. It is about data. Specifically, how we label it, tag it, and standardize it across the enterprise so our future AI systems can actually learn, operate, and make decisions. At SOCPAC, we learned this lesson the hard way. You cannot scale autonomy unless you first standardize the data that feeds the models. Here is why a Department-wide Data Standardization is no longer optional. It is the prerequisite for unlocking LLM-enabled planning assistants, computer vision-based targeting systems, autonomous UxS swarms, and resilient multi-agent operations. The way I see it, the models and autonomous systems are just commodities, and they are only as useful as our data is organized. Why Labeling and Metadata Tagging Matter LLMs and CV models do not learn from raw data. They learn from structured, labeled, and standardized data. If every organization uses different schemas and naming conventions, the models cannot generalize. If metadata is missing or inconsistent, autonomous systems cannot reason with confidence. As you can imagine Industry has figured this out years ago. Every high-performing AI company has a central data governance function responsible for: • Global taxonomies and data dictionaries • Unified metadata schemas • Automated labeling pipelines • Standardized APIs for every system • Platform-independent data transport layers The Public Sector has to adopt these practices now. Not in 2030. The Bottom Line We cannot build autonomous UxS fleets, multi-agent coordination systems, or LLM-driven workflows without clean, labeled, standardized data flowing through open interfaces. Data standardization is national defense.

The Critical Role of Metadata in Agentic Systems 🛄 Metadata serves not merely as a supplementary component but as the essential foundation that enables AI agents to function effectively within enterprise environments. Metadata's role has evolved from basic data description to becoming the crucial infrastructure that powers sophisticated AI agents. First, metadata serves as the cognitive foundation for AI agents, enabling four critical capabilities: reasoning support, external memory enhancement, execution capabilities, and planning functions. Through the integration of descriptive, structural, administrative, and semantic metadata, agents can understand context, maintain knowledge structures, and make informed decisions. This multi-layered approach to metadata management creates a comprehensive framework that supports increasingly sophisticated AI operations. Second, the convergence of metadata management with LLM Mesh Architecture demonstrates how metadata facilitates the creation of distributed, specialized AI agents while maintaining system coherence. Metadata plays a crucial role in enabling agent communication, task orchestration, and knowledge sharing across distributed systems. This is particularly evident in how semantic metadata creates bridges between different domain-specific agents and their respective knowledge bases. Third, metadata's role in enterprise settings extends beyond technical implementation to encompass governance, compliance, and scalability considerations. Organizations must view metadata management as a strategic imperative, particularly as AI agents become more autonomous. Administrative metadata ensures appropriate access controls and audit capabilities, while semantic metadata enables agents to operate within defined business constraints. The depth of metadata's importance becomes apparent when examining how it enables advanced agent capabilities. Metadata creates a semantic layer that allows agents to understand not just the structure of data, but its meaning and relationships. This semantic understanding is crucial for agents to make context-aware decisions and adapt to new situations. Furthermore, the integration of different metadata types creates a comprehensive framework that supports both operational efficiency and governance requirements. The evolution of software architectures towards distributed agentic systems, places increasing demands on metadata management. As systems become more distributed and autonomous, metadata must evolve to support more sophisticated interaction patterns and knowledge structures. This evolution is evident in the emergence of hybrid approaches that combine traditional metadata management with AI-powered capabilities. Metadata serves as a bridge between traditional enterprise systems and emerging AI capabilities, particularly in how it enables the creation of sophisticated knowledge graphs and ontologies that power AI agent decision-making.

I have spent more than 20 years building and operating open source data infrastructure. I have seen the Hadoop era, the rise of streaming, Kafka at massive scale, cloud data platforms, governance platforms, and now AI. Every era had a central question. In big data, the question was: can we store and process all this data? In streaming, the question was: can we react in real time? In cloud, the question was: can we scale elastically? In AI, the question is becoming: can we trust the context behind the answer? That is why I believe metadata is becoming one of the most important layers in the enterprise AI stack. AI needs to know what data means, where it came from, who owns it, whether it is trusted, how it changed, and what policies apply to it. Without that context, AI is just guessing over poorly understood data. With that context, AI can help people make better decisions. This is the space I am spending my time for past 5 years: openmetadata, governance, semantics, lineage, ontologies, and AI-ready data platforms. My view is simple: The future of enterprise AI will be built on metadata.

Documentation used to be viewed as overhead. Now it looks a lot more like infrastructure. At TRM Labs, we’ve long believed that writing down decisions, definitions, and assumptions is part of how distributed teams move quickly. What’s become clearer recently is that this habit also makes AI systems work better. In our latest blog post, we investigated the value of documentation in improving performance for AI agents. Documented agents achieved 100% agreement on a simple analytics query and 99% on a complex one with the same prompt. Without documentation, agreement dropped to 36% and 15%. That’s the difference between “the model can use tools” and “the model can reliably do useful work.” The interesting part is that prompt phrasing mattered less than many people expect. For simple questions, well-documented data held up across 20 different wordings. For complex questions, the bigger problem was hidden ambiguity. So the playbook is becoming clearer: - invest in metadata and documentation - let agents inspect the right tools - use critic / challenger patterns when assumptions matter 👉🏽 https://lnkd.in/gNstwDMv

Metadata explained: Metadata is one of those terms people throw around a lot but rarely define clearly. Let’s bring some light into the role it actually plays in real-world data systems.   Metadata is the quiet backbone of good data platforms.   Not the data itself, but the information around it. When something was processed. How much. By which logic. With what structure.   It’s easy to overlook, especially early on. Because in the beginning, you just want the pipelines to “work.”   You build a pipeline, you see the data show up in your warehouse, and you move on. (It's a bit of an exaggeration, but I think you know what I mean)   Here’s the trap: If you don’t collect metadata now, you’ll end up flying blind later.   Debugging becomes guesswork. Audits become painful. You start chasing errors.   𝗜𝗳 𝗜 𝗰𝗼𝘂𝗹𝗱 𝗴𝗼 𝗯𝗮𝗰𝗸 𝗮𝗻𝗱 𝘁𝗲𝗹𝗹 𝗺𝘆 𝗽𝗮𝘀𝘁 𝘀𝗲𝗹𝗳 𝗼𝗻𝗲 𝘁𝗵𝗶𝗻𝗴 𝗮𝗯𝗼𝘂𝘁 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗱𝗮𝘁𝗮 𝘀𝘆𝘀𝘁𝗲𝗺𝘀, 𝗶𝘁 𝘄𝗼𝘂𝗹𝗱 𝗯𝗲 𝘁𝗵𝗶𝘀: 👉 “Start collecting metadata, even when it feels unnecessary.”   Because eventually things will break. And when they do, you’ll want more than logs. You’ll want to understand what happened.   Some of the metadata fields that saved us more than once: ✅ When did the job run, and how long did it take? ✅ How many records were read, written, skipped, or failed? ✅ Was the schema version what we expected? ✅ Did key fields go missing or suddenly become null? ✅ What config or code version triggered the run?   This kind of metadata isn't “nice to have.” It’s how you make your system observable, especially when it's distributed, automated, and touched by 10+ other processes.   And the best thing is if you have all this information in one place to view it in one dashboard, one tool.   You won’t check these metrics daily. But when someone asks “What went wrong?” or “Can we trust this?”, metadata is the only thing that can answer with confidence.   👉 Think of it as observability with hindsight built in. 👉 Think of it as your future self’s best friend.

Four years ago, I made a bet. Metadata would become one of the most important layers in the modern data stack. Turns out, the bet was too small. Metadata didn't just become important for the data stack. It became the foundation of enterprise AI. Every company doing serious work with AI agents hit the same wall. Not a model problem. Not an infrastructure problem. A context problem. The people who maintained your metadata, the column descriptions, lineage maps, business glossaries, often thanklessly, turned out to be building the most important layer in the enterprise AI stack. They just didn't have a name for it yet. Last June, Andrej Karpathy gave it one: context engineering. Tobias Lütke called it the core skill set he'd been waiting to see named. By Gartner's D&A Summit 2026, it stopped being a term and became a theme. IBM, a16z, Anthropic all published on it. But this community saw it first. The writing got here first. The name follows. Metadata Weekly is becoming Context and Chaos (& Cats). Same community. Same practitioner-first lens. More to cover, because right now there's a window. The people who've been doing this work for years have the credibility and knowledge to shape what context engineering becomes. I'd rather this community write the definitions than watch from the sidelines. First issue under the new name drops next week. 👇 Read the full edition and see what's new with Context & Chaos. We think you'll love it!