Metadata Management

For Data Engineers, Databricks Unity Catalog is the secret to managing data at scale across teams, clouds, and projects. But what exactly is behind that term?   𝗨𝗻𝗶𝘁𝘆 𝗖𝗮𝘁𝗮𝗹𝗼𝗴 is the unified governance layer for 𝗮𝗹𝗹 your data and assets. Tables, files, notebooks, ML models, you name it.   It’s not just another feature; it’s a 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 for managing your data platform at scale, making data usable, secure, and trustworthy across your whole platform.   Here’s the core idea: - 𝗠𝗲𝘁𝗮𝘀𝘁𝗼𝗿𝗲: A single source of truth for metadata. One per region, no more per-workspace metastores. - 𝗖𝗮𝘁𝗮𝗹𝗼𝗴𝘀: Group your data by business domains → sales, marketing, operations. - 𝗦𝗰𝗵𝗲𝗺𝗮𝘀: Organize data logically → Bronze, Silver, Gold layers. - 𝗧𝗮𝗯𝗹𝗲𝘀 & 𝗩𝗶𝗲𝘄𝘀: Where your data lives. Structured, secure, discoverable. - 𝗘𝘅𝘁𝗲𝗿𝗻𝗮𝗹 𝗟𝗼𝗰𝗮𝘁𝗶𝗼𝗻𝘀: Securely link cloud storage with access policies via Storage Credentials.   But it’s not just structure. Here’s what Unity Catalog really brings to the table: ➡️ 𝗖𝗲𝗻𝘁𝗿𝗮𝗹𝗶𝘇𝗲𝗱 𝗮𝗰𝗰𝗲𝘀𝘀 𝗰𝗼𝗻𝘁𝗿𝗼𝗹: Manage permissions across all Databricks workspaces. No more messy, scattered permission settings. ➡️ 𝗙𝗶𝗻𝗲-𝗴𝗿𝗮𝗶𝗻𝗲𝗱 𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆: Control access down to the column or even row level. Perfect for sensitive data (PII, anyone?). ➡️ 𝗠𝘂𝗹𝘁𝗶-𝗳𝗼𝗿𝗺𝗮𝘁 𝘀𝘂𝗽𝗽𝗼𝗿𝘁: Delta, Iceberg, Hudi. Work with the formats your team needs, no vendor lock-in. ➡️ 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗹𝗶𝗻𝗲𝗮𝗴𝗲: See exactly how data flows: from Bronze to Silver to Gold. Great for debugging, impact analysis, and compliance. ➡️ 𝗘𝗻𝗱-𝘁𝗼-𝗲𝗻𝗱 𝗹𝗶𝗻𝗲𝗮𝗴𝗲: Trace data from ingestion to final report, automatically updated in real time. and more...   For Data Engineers, Unity Catalog means fewer headaches and more confidence: - No more separate metastores per workspace. - Clear separation of storage (where data lives) and metadata (who can access what, how, and why). - Full traceability of every transformation, whether it’s a small type cast or a complex data model change.   ***** In my Azure Databricks project, we put Unity Catalog into practice! Here, you'll: ➡️ Set up storage credentials to access Azure Data Lake securely. ➡️ Create external locations for raw and processed data. ➡️ Organize data in a 3-level namespace: catalog.schema.table → aligned with Medallion Architecture. ➡️ Control access: Business users only see Gold, engineers get access to Silver/Bronze for transformation.   This is how we make complex data systems manageable. And this is what modern Data Engineers need to build.   🎓 Want to learn how it works, step by step? Check the project link in the comments! 👇

This paper describes how a large pharmaceutical company adopted an ontology-based data management strategy to ensure scientific data is findable, accessible, interoperable, and reusable from the moment it is generated. 1️⃣ The approach emphasizes creating structured, high-quality data at the source to preserve context and reduce downstream processing time. 2️⃣ Standardized vocabularies and models (ontologies) are used to align data across systems and teams, supporting consistency and integration. 3️⃣ Public ontologies are adapted with organization-specific extensions while maintaining compatibility with external data standards. 4️⃣ Simplified term lists are derived from complex models to enable broader adoption across teams with varying technical backgrounds. 5️⃣ Data from different systems is integrated virtually rather than physically moved, enabling secure, real-time access without redundancy. 6️⃣ This framework enhances the performance of advanced analytics and machine learning by providing clear, semantically rich context. 7️⃣ Controlled vocabularies are delivered through interfaces like APIs and dropdowns, ensuring consistent metadata usage at scale. 8️⃣ The unified semantic structure improves enterprise search, allowing users to retrieve contextually relevant data from across domains. 9️⃣ Adoption metrics show growing usage across multiple phases of the pharmaceutical value chain, reflecting system scalability and value. 🔟 Organizational alignment—from executive support to operational implementation—has been critical, with recent advances in AI further enabling this transformation. ✍🏻 Shawn Zheng Kai Tan, Shounak Baksi, Thomas Gade Bjerregaard, Preethi Elangovan, Thrishna Kuttikattu Gopalakrishnan, Darko Hric, Joffrey Joumaa, Beidi Li, Kashif Rabbani, Santhosh Kannan Venkatesan, Joshua Daniel Valdez, Saritha Vettikunnel Kuriakose, Digital evolution: Novo Nordisk’s shift to ontology-based data management. Journal of Biomedical Semantics. 2025. DOI: 10.1186/s13326-025-00327-4

This image illustrates how I’m thinking about metadata/ontologies/knowledge graph/semantic layers Left: we have the “Governed Metadata” which contains governed business, technical, and mapping metadata.  1️⃣ Business Metadata: Your glossaries, taxonomies, ontologies. The shared language of the business. 2️⃣ Technical Metadata: Schemas, tables, columns, data types. Extracted directly from systems like relational databases. 3️⃣ Mapping Metadata: this is the bridge that connects the technical to business metadata. It’s where meaning (i.e. semantics)  happens. These three parts evolve independently (and often do). Governance is how this gets aligned otherwise this turns into a “boiling the ocean”. Together, they form the core of your enterprise brain, the metadata foundation that gives your data context, structure, and meaning. Right: AI requires context and that is why it is driving the demand for Knowledge Graphs and BI Semantic Layers. Each tool expects metadata in its own syntax or format because it is dependent on the deployment mechanism of each tool. That is why I’m calling this “Deployed Metadata”, because it represents tool-specific, executable outputs like YAML, etc. Middle: we have a “Metadata Deployment Engine” which takes the governed metadata and transforms it into the syntaxes/formats specific to downstream platforms and tools. This is what takes the governed metadata and pushes out versions to each of these downstream systems consistently. The real power: ✅ Define and Governance once ✅ Deploy anywhere ✅ Stay aligned across tools. This is how we avoid having multiple answers for the same question What should power the Governed Metadata? My position: it should be a graph, and more specifically, RDF, because: - RDF is an open web standard made to connect resources - Supports ontologies (OWL), taxonomies (SKOS), validations (SHACL), provenance (PROV), etc - Built for reuse, governance, and interoperability of metadata across systems (the Web is the largest system!) 1️⃣ Business Metadata :OrderLineItem a owl:Class ;   rdfs:label "Order Line Item" . :OrderLineItemQuantity owl:DatatypeProperty ;   rdfs:label "Order Line Item Quantity" ;   rdfs:domain :OrderLineItem ;   rdfs:range xsd:int. 2️⃣ Technical Metadata :lineitem a dw:Table ;   dw:hasColumn :quantity . :l_quantity a dw:Column ;   dw:dataType "DECIMAL(15,2)" ;   dw:isNullable true . 3️⃣ Mapping Metadata :l_quantity dw:represents :OrderLineItemQuantity . :lineitem dw:represents :OrderLineItem . If you aim to support rich, linked, governed metadata across systems, and you don’t use RDF... you're probably going to end up building something like RDF anyway… just less standardized, less interoperable, and harder to maintain. As Mark Beyer states, "metadata is a graph", and that is why data catalog and governance platforms should be on a knowledge graph architecture. I plan to share more sophisticated examples next, but wanted to get this out first and see how folks react.

Metadata is King 🔥. Treat it like one! Well, actually metadata is big data - that was the core thesis of this foundational paper (VLDB 2021) from the Google BigQuery team. This was published at a time when cloud data warehouses were scaling to petabytes of data. The challenge? ⛔️ As BigQuery scaled to petabyte-scale tables with billions of blocks and 10K+ columns, their metadata itself became a performance bottleneck. ⛔️ Reading metadata from file footers didn’t scale & centralized approaches couldn’t keep up with interactive query demands. ⛔️ Most queries only touch a small subset of columns & often scan less than 0.01% of data. Without efficient metadata pruning, even these light queries ended up paying a heavy cost. So, the BigQuery team approached this slightly differently. They asked: What if we treated metadata the same way we treat data itself? The result was a fully distributed, columnar metadata system (CMETA) that could scale to tens of TBs of metadata, power adaptive query planning, and dramatically cut down on latency and resource usage. Instead of centralized catalogs, BigQuery: - Stores metadata in a columnar internal table (CMETA) with block-level stats (min/max, bloom, dictionary, etc.) - Uses the same distributed Capacitor format as regular data tables for column pruning & parallelism - Uses falsifiable expressions to eliminate irrelevant blocks before execution - Defers metadata resolution to runtime, enabling adaptive query planning - Supports time travel, incremental mutation tracking, and streaming updates with ACID guarantees Pretty much like building any data storage system right? This resulted in: - 30,000x lower resource usage for selective queries over 1PB tables - 50x faster queries by avoiding unnecessary block scans That's not the end. Modern day #lakehouse systems like Apache Hudi takes huge inspiration from this. To tackle the scaling challenges with large amount of metadata, Hudi introduced a dedicated internal "metadata table" that mirrors many of the same principles: In Hudi: ✅ A Merge-On-Read internal metadata table tracks file listings, column stats, bloom filters, and more. ✅ Stored in the same Hudi format (MoR), enabling scalable and transactional updates ✅ Partitioned by metadata types (file listing, column stats, etc.) ✅ Uses HFile-based base files (SSTable style) for fast column-specific lookups ✅ Can be served via embedded timeline server for ultra-low latency reads The takeaway is that - as data systems scale, metadata becomes the first-class citizen. So, how we manage it impacts overall cost & performance of queries. Paper link & Hudi docs in comments! #dataengineering #softwareengineering

AI needs a different kind of data management to succeed—letting go of neatly structuring the world and doubling down on metadata. For decades, corporate data management meant discipline: defined fields, taxonomies, and carefully crafted data models built to impose order. Information was captured, cleansed, and contained under the assumption that insight followed structure. That assumption no longer holds for today's AI systems (=LLMs and other generative models). They have little use for traditional tidiness. They train and infer not on tabulated records but on vast, unruly troves of unstructured content. What matters isn't order, but abundance, diversity, and context. Yet many organizations still treat AI as sophisticated BI or supervised ML. Investments flow into rigid structures and polished pipelines pursuing "AI-ready data." But AI isn't a more fancy dashboard. It serves a different purpose: learning patterns in highly unstructured data and dealing with ambiguity. Therefore, AI needs a different kind of data management that shifts from enforcing structure to enabling understanding. If unstructured data is AI's raw material, then metadata—the data about the data—is its essential scaffolding. In a world where AI trains on noise, metadata provides the signal. It identifies sources, flags permissions, captures provenance, encodes trust. It tells systems not just what content is, but who created it, in what context, and how credible it might be. It helps models distinguish satire from sincerity, guidance from opinion, sensitive from shareable. Data quality is fundamental as much as ever, but in a different way. Yes, AI is vulnerable to biases and factual errors. But fixing this hinges less on conformity to schemas and more on richness, representativeness, and reliability. Metadata becomes critical where AI meets legal, ethical, and regulatory demands: access controls, lineage, consent, auditability—these depend not on content structure, but on surrounding metadata, enabling responsible use of messy data. If unstructured content is the terrain, metadata is the map. The task isn't abandoning data management, but evolving it. Structured systems remain vital for transactions, but AI's promise lies in embracing the richness—and mess—of the real world, while building tools to navigate it wisely. Organizations that thrive in the AI era won't be those with the cleanest data warehouses, but those with sophisticated metadata ecosystems. This shift from data hygiene to data context represents not just a technical evolution, but a philosophical one—acknowledging that in a complex world, understanding often matters more than order.

𝗗𝗮𝘁𝗮 𝗴𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗶𝘀 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 𝗺𝗶𝘀𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗼𝗼𝗱 𝘁𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗲𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲. Because most people explain it from the inside out: policies, councils, standards, stewardship. But the business does not buy any of that. The business buys outcomes: → trustworthy KPIs → vendor and partner data you can actually use → faster financial close → fewer reporting escalations → smoother M&A integration → AI you can deploy without creating risk debt Most AI programs fail for boring reasons: nobody owns the data, quality is unknown, access is messy, accountability is missing. 𝗦𝗼 𝗹𝗲𝘁’𝘀 𝘀𝗶𝗺𝗽𝗹𝗶𝗳𝘆 𝗶𝘁. 𝗗𝗮𝘁𝗮 𝗴𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗶𝘀 𝗳𝗼𝘂𝗿 𝘁𝗵𝗶𝗻𝗴𝘀: → ownership → quality → access → accountability 𝗔𝗻𝗱 𝗶𝘁 𝗯𝗲𝗰𝗼𝗺𝗲𝘀 𝘃𝗲𝗿𝘆 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝘄𝗵𝗲𝗻 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸 𝗶𝗻 𝟰 𝗹𝗮𝘆𝗲𝗿𝘀: 1. Data Products (what the business consumes) → a named dataset with an owner and SLA → clear definitions + metric logic → documented inputs/outputs and intended use → discoverable in a catalog → versioned so changes don’t break reporting 2. Data Management (how products stay reliable) → quality rules + monitoring (freshness, completeness, accuracy) → lineage (where it came from, where it’s used) → master/reference data alignment → metadata management (business + technical) → access controls and retention rules 3. Data Governance (who decides, who is accountable) → data ownership model (domain owners, stewards) → decision rights: who can change KPI definitions, thresholds, and sources → issue management: triage, escalation paths, resolution SLAs → policy enforcement: what’s mandatory vs optional → risk and compliance alignment (auditability, approvals) 4. Data Operating Model (how you scale across the enterprise) → domain-based setup (data mesh or not, but clear domains) → operating cadence: weekly issue review, monthly KPI governance, quarterly standards → stewardship at scale (roles, capacity, incentives) → cross-domain decision-making for shared metrics → enablement: templates, playbooks, tooling support If you want to start fast: Pick the 10 metrics that run the business. Assign an owner. Define decision rights + escalation. Then build the data products around them. ↓ 𝗜𝗳 𝘆𝗼𝘂 𝘄𝗮𝗻𝘁 𝘁𝗼 𝘀𝘁𝗮𝘆 𝗮𝗵𝗲𝗮𝗱 𝗮𝘀 𝗔𝗜 𝗿𝗲𝘀𝗵𝗮𝗽𝗲𝘀 𝘄𝗼𝗿𝗸 𝗮𝗻𝗱 𝗯𝘂𝘀𝗶𝗻𝗲𝘀���, 𝘆𝗼𝘂 𝘄𝗶𝗹𝗹 𝗴𝗲𝘁 𝗮 𝗹𝗼𝘁 𝗼𝗳 𝘃𝗮𝗹𝘂𝗲 𝗳𝗿𝗼𝗺 𝗺𝘆 𝗳𝗿𝗲𝗲 𝗻𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: https://lnkd.in/dbf74Y9E

As a data engineer, I was more interested in building pipelines and solving tech puzzles, not setting up policies and processes. Little did I realize that data governance was the backbone of the very systems I relied on. Fast forward to today, and my perspective has completely shifted. Working on an entire data platform taught me that data governance is more than rules and restrictions; it’s the glue that holds everything together. Think of it as being the GPS for your organization’s data—it helps you navigate, keeps your data secure, and ensures everyone reaches their destination smoothly. I started seeing data governance as essential when I faced real-world problems: ▪️ Reports built on inaccurate data. ▪️ Duplicate or missing records causing business losses. ▪️ Sensitive information being exposed due to improper controls. It became clear that governance wasn’t an optional add-on; it was the foundation for ensuring trust in the data. So, What is Data Governance? It’s like onboarding a new employee. Just as every new hire is introduced to the company’s policies and trained for their role, every piece of data needs rules and a structure to follow. This ensures: ▪️ The data is high-quality and trustworthy. ▪️ It’s accessible only to the right people. ▪️ It’s traceable, so you know where it came from and how it’s been used. Here’s how I like to explain the main aspects of data governance: 1. Metadata Management Imagine a treasure map where the “X” marks the data you need. Metadata is that map. It tells you what the data represents, its origin, and how to use it effectively. Without it, you’re just guessing in the dark. 2. Data Access Control Think of a vault in a bank. Not everyone gets the same key. Permissions are granted based on roles, ensuring sensitive data stays protected while authorized users get what they need. 3. Data Lineage Ever traced a package you ordered online? Data lineage works the same way. It tracks where the data came from, where it’s going, and what’s been done to it. This visibility ensures accuracy and helps fix issues faster. 4. Data Access Audit This is your security camera. It logs who accessed what and when, providing a trail that keeps the system secure and compliant. 5. Data Discovery Finally, imagine a search engine for your organization’s data. It helps you find the exact dataset you need, fostering innovation and smarter decisions. So, next time you think of governance as just red tape, remember: It’s the invisible infrastructure making everything else work smoothly. The cleaner and safer your data, the more power it holds. What’s your take on data governance? Have you faced any challenges or successes with it? ❣️Love it...spread it ♻️

I feel that the most underestimated pillar of Databricks Unity Catalog implementation in any organisation is the metadata design and stewardship. Generally speaking, when people talk about Unity Catalog, most part of the conversation revolves around centralised governance, fine-grained access controls, and the shiny promise of a single data security model. Yet, in my opinion, one critical component rarely gets the attention that it deserves - metadata design. 💡 I tried to understand in depth why it's often overlooked... 1️⃣ Perceived as "Documentation": Teams often treat metadata as an afterthought. It's something that can be added (later) after ingestion pipelines are built. 2️⃣ Lack of Ownership: Generally, metadata doesn't fall neatly under engineering, governance, or business teams, so it becomes a "shared" blind spot. 3️⃣ Tooling Gaps: While UC supports rich metadata, there's a lack of plug-and-play tooling that pushes teams to input this info early and keep it updated. Sometime, even having multiple metadata tooling also doesn't help due to integration issues. 🔥 The hard way of learning things... While building enterprise-scale data platforms, poor metadata can have severe impacts: 1️⃣ Data Discoverability: Without proper descriptions and tags, even authorised users struggle to find relevant datasets. Imagine what happens when you have thousands of datasets sitting live in prod. 2️⃣ Data Quality Perception: You (as a user) can't trust what you don't understand. 3️⃣ Automation Readiness: The AI foundation integrations rely on high-quality, structured metadata. That's why, in my opinion, we should start treating "Metadata As Code" - well defined, versioned, and validated alongside UC objects. 💪 But how do you elevate metadata in Unity Catalog ⁉️ 1️⃣ Just set the metadata standards from day one: Define conventions for naming, descriptions (table/volume & column), tags, and ownership. Enforce them via backend automation. 2️⃣ Automate where possible: We talk a lot about data pipeline automation. Now, it's time to think of metadata. Considering you have hundreds of tables live in production, it's very easy to build an internal ML model (I've tried it using XGBoost) that gets trained on existing metadata and helps bootstrap suggested column data types and descriptions based on schema. 3️⃣ Data Contracts: Metadata stewardship can be made part of dev and review lifecycle, just like code reviews, involving both data producers and consumers. 4️⃣ Tag usage: Use tags strategically, especially for data classification (PII, Mission-Critical, etc.) lifecycle, and SLA indications. 5️⃣ Monitor metadata drift: Just like data drift, stale metadata can make your dataset outdated resulting in loss of trust. UC System Tables (information_schema.tables) can be used to build an observability around it. "Somebody" told me.. if Unity Catalog is your control plane, then metadata is the UI. 🙂 #Databricks #UnityCatalog #DataGovernance #DataPlatform

Copilot just learned to read your custom metadata. wait what! This changes everything about SharePoint your governance. Until now, metadata was mostly about search and compliance. Nice to have. But hard to justify the effort, let's be honest. But as of right now, Copilot reasons over your custom SharePoint properties. Equipment specs? It reads make, model, engine size. Project docs? It understands status, owner, phase. Compliance files? It knows approval dates, reviewers, categories. Not generic summaries. Precise answers grounded in YOUR business context. Here's why I think this is really important for you... Metadata used to be a filing system. Now it's how Copilot understands your business. The effort you put into structured properties, content types, and taxonomies? That directly determines how useful Copilot becomes. Strong metadata = Copilot that knows your context. Weak metadata = Copilot that might just guess. Organizations that invested in metadata early are already seeing the difference. Precise answers. Faster adoption. Automated governance. Organizations that skipped it are realizing metadata just became business critical.

Is Master Data Management (MDM) still relevant in an AI era? What's the future of MDM? The short answer to the first question is 'absolutely yes', and here's why: As long as our business functions have distinct vocabularies and have flexibility to maintain their own governance policies, there will be a need for rules-based processes to resolve those differences. GenAI is exceptionally good at understanding languages, but it's not good at following rules that are explainable, repeatable, and consistent. That's why there remains a role for MDM now, and for the foreseeable future. Speaking of the future... MDM is a critical foundation of a data estate, but it must necessarily evolve. Companies must acknowledge that the concept of 'truth' is contextually bound, and that *multiple* versions of truth necessarily exist in all organizations. MDM software can support this today - but governance programs must adapt to more widely embrace the reality that the way that marketing looks at the world is different than the way finance looks at the world. The future of MDM also requires the discipline to drastically expand the volumes and variety of data within its scope. MDM will serve as the beating heart of an evolving semantic layer that spans the realms of data management, information management, and knowledge management. MDM and other metadata management tools, like data catalogs, will increasingly converge into a unified platform that will provide the context and meaning needed for both downstream operational systems and advanced analytics platforms, like AI. The critical difference which will separate MDM from other capabilities within this evolving semantic layer is the ability to enforce data governance policies at the level of an 𝐢𝐧𝐝𝐢𝐯𝐢𝐝𝐮𝐚𝐥 𝐫𝐞𝐜𝐨𝐫𝐝. Ontologies, taxonomies, and dictionaries will virtually connect concepts, objects, and entities within this semantic layer - but MDM will support this entire foundation by ensuring what's represented at a record level is accurate, consistent, and trustworthy. To do this, MDM programs must necessarily expand into the realms of unstructured data, where data discovery and profiling tools will help MDM programs to understand exactly which data should, or shouldn't, be included within the scope of an MDM program. Other capabilities in this evolving semantic layer will bring more structure to this data - so the rules managed within an MDM can be applied at scale across an ever-increasing volume of data. AI will most certainly assist in the processes of discovery, tagging, and structuring data, as it will also help to significantly expand the throughput of data stewards - which will remain a necessary ingredient for many use cases. AI is changing the game, but our need to support the core data management fundamentals - like MDM - has not. What do you think? #mdm #masterdatamanagement #datagovernance