User Experience Metrics for Success

Here's 7 key metrics every AI PM should track — not just to measure engagement, but to ensure your AI is useful, safe, and trusted. Too often, we focus on DAU or churn… but, especially if you're building conversational products, you need new metrics — ones that capture meaning, depth, and trust. Here’s the framework I use 👇 I used a pyramid because each layer supports the next: without factual, safe foundations, you can’t earn trust or scale responsibly. -The foundation is Model Quality — your AI must be accurate, safe, and fast before anything else matters. -Above that is Interaction Quality — can users have meaningful, multi-turn conversations that feel natural and helpful? -Then comes Trust & Delight — do users enjoy the experience and come back because they trust it? -Higher still is User Value — are people actually achieving their goals faster, easier, and better? - And at the top sits Sustainability — are you doing all of this responsibly and efficiently (revenue / compute $, LTV / CAC)? Success in conversational AI = Useful × Safe × Trusted <><><><><><><><><><> Follow Marily Nika, Ph.D for AI PM education, certifications and insights.

🔥 If your Data Catalog isn’t measured, it’s probably failing. Most data catalogs don’t fail because of technology. They fail because success is never clearly defined. So let’s be blunt. Here’s how you actually know whether your data catalog works. ❌ Vanity metric to forget: “Number of datasets cataloged” ✔️ Metrics that matter: 🔴 1. Do people come back? (Adoption) One login ≠ success. Are users still active after onboarding? Are they searching… or asking Slack instead? If usage drops, your catalog is just expensive documentation. 🔴 2. Is the metadata good enough to trust? Auto-ingested metadata ≠ usable metadata. Do datasets have owners? Are descriptions written for humans? No context = no trust = no usage. 🔴 3. Does it actually save time? If analysts still spend hours “data hunting”, the catalog failed. Can users find the right dataset in minutes? Are the same questions still asked every week? If nothing changes, value is zero. 🔴 4. Who is accountable for the data? “Shared responsibility” usually means “no responsibility”. Is every critical dataset owned? Do stewards respond? Governance starts with naming names. 🔴 5. Can users tell which data is safe to use? Without trust signals, catalogs create confusion — not clarity. Certified datasets Data quality visibility Clear warnings for risky data No signals = no confidence = shadow data. 🔴 6. Is the platform reducing manual effort — or creating more? If stewardship feels like extra work, it won’t scale. How much is automated? Is steward workload increasing or decreasing? If governance doesn’t scale, it dies. 🔴 7. Does the business feel the impact? This is the uncomfortable question. Faster decisions? More reuse? Fewer duplicated datasets? If leadership can’t feel the difference, they won’t fund it. ⚠️ Hard truth: A data catalog is not a compliance tool. It’s not a metadata repository. It’s not a checkbox. It’s a product, and products live or die by adoption, trust, and impact. 💬 Be honest: Which of these KPIs are you actually tracking today?

Ever looked at a UX survey and thought: “Okay… but what’s really going on here?” Same. I’ve been digging into how factor analysis can turn messy survey responses into meaningful insights. Not just to clean up the data - but to actually uncover the deeper psychological patterns underneath the numbers. Instead of just asking “Is this usable?”, we can ask: What makes it feel usable? Which moments in the experience build trust? Are we measuring the same idea in slightly different ways? These are the kinds of questions that factor analysis helps answer - by identifying latent constructs like satisfaction, ease, or emotional clarity that sit beneath the surface of our metrics. You don’t need hundreds of responses or a big-budget team to get started. With the right methods, even small UX teams can design sharper surveys and uncover deeper insights. EFA (exploratory factor analysis) helps uncover patterns you didn’t know to look for - great for new or evolving research. CFA (confirmatory factor analysis) lets you test whether your idea of a UX concept (say, trust or usability) holds up in the real data. And SEM (structural equation modeling) maps how those factors connect - like how ease of use builds trust, which in turn drives satisfaction and intent to return. What makes this even more accessible now are modern techniques like Bayesian CFA (ideal when you’re working with small datasets or want to include expert assumptions), non-linear modeling (to better capture how people actually behave), and robust estimation (to keep results stable even when the data’s messy or skewed). These methods aren’t just for academics - they’re practical, powerful tools that help UX teams design better experiences, grounded in real data.

One of the key ways to demonstrate the value of UX research is by measuring success metrics. Without these, it can be hard to show the impact of your work on the product or the business. But how exactly can we measure success in a UX research project? Here are a few critical steps and metrics to consider: 1. Align with Business Goals: ↳ Start by identifying the KPIs tied to business goals. Whether it’s conversion, adoption, or drop-off rates, the research should connect to metrics that matter for the company’s success. By linking research insights directly to business outcomes, you show stakeholders how UX impacts their key priorities. 2. Behavioral Metrics: These are the data points tied to how users interact with your product, such as: ↳ Task Success Rate: How many users successfully complete the task? ↳ Time-on-Task: How long does it take users to complete a task? ↳ User Error Rate: How often do users make mistakes during the task? Tracking these helps identify friction points in the user journey and quantifies the effectiveness of your designs. 3. Attitudinal Metrics: These reflect how users feel about the product or experience: ↳ Net Promoter Score (NPS): How likely are users to recommend your product? Although this one is definitely not my favorite, most businesses care a lot about NPS. ↳ Customer Satisfaction (CSAT): How satisfied are users with the product? ↳ Perceived Ease of Use: How easy do users think the product is to use? Gathering these insights gives you a clear sense of user sentiment and overall satisfaction. 4. Usability Metrics: For more specific insights, you can track usability metrics like: ↳ System Usability Scale (SUS): A quick way to assess perceived usability. ↳ Completion Rates: How many users completed a given task without assistance? 5. Impact on KPIs: Finally, after research is complete and changes are implemented, re-measure these metrics to show improvements. Demonstrating a reduction in error rates or an increase in task success ties UX research directly to improved product performance. By clearly connecting UX metrics to business KPIs, you help stakeholders see the concrete value that research brings to the table. These success metrics aren’t just numbers — they’re proof of how UX research improves user experience and drives business impact. How do you measure success in your UX research projects?

Most people have no clue how to effectively measure the success of their data products, let alone data more broadly. (And yes, by 'data products' I mean everything from APIs, analytics dashboards, and AI models to traditional datasets.) If you're managing data products (officially or unofficially), you've got to get serious about KPIs. Not only does it create focus and prioritization for data and governance teams, it proves the value of why you are doing a data product approach in the first place. Some quick, honest no-BS thoughts: 📈 Adoption & Usage: Is anyone actually using your data product? Track unique users, frequency, API calls, dashboard views, queries run. ⚙️ Reliability & Quality: Is your product trustworthy? Measure downtime, response times, data accuracy, freshness, and error rates. 💰 Business Impact: Are you actually making the company money or saving it? Look at revenue generated, costs reduced, efficiency gained, or strategic decisions enabled. 🎭 User Satisfaction: Do users like (or hate) your data product? Monitor NPS, satisfaction surveys, feedback loops, and direct user comments. 🛠️ Scalability & Maintainability: Can you sustain growth without headaches? Consider tech debt, scalability metrics, and cost of operations. Metrics mean nothing though if they're not driving decisions. Don't just track; ACT: ➡️ If adoption is low, reconsider product-market fit. ➡️ High error rates? Invest in quality and testing. ➡️ Positive business impact? Double down and expand. Do you have certain KPIs do YOU find most valuable for your data products? Or data investments more generally? #DataProducts #DataProductManagement #DataStrategy #BusinessValue #DataGovernance #DataLeadership #KPIs

Not all data quality dimensions can be validated with structural rules like “value is not null” or “format is correct.” Some dimensions require completely different measurement approaches. One such example is Data Usability. Many practitioners struggle with how to measure for Data Usability. Usability is not about whether data passes technical validation—it’s about whether people can actually find it, trust it, understand it, and use it without friction. Usability lives in the experience of the data consumer. That’s why it can be measured through data consumer surveys and questionnaires. We can ask questions such as: - “How easy is it to find the data you need?” - “Do you trust this dataset for decision-making?” - “Is the data understandable without additional interpretation?” - “Can you use it without significant transformation or cleanup?” But even more powerful are behavioral signals that reveal usability in practice. For example, if analysts consistently rely on their own Excel extracts instead of governed datasets, that is a clear usability red flag. Or if a new data platform (e.g., Snowflake) is available, but business users continue using an older SQL Server database built 20 years ago, that is a strong signal that the newer data is not truly usable. These behavioral patterns often tell us more than any rule-based validation ever could. This post was inspired by the Data Stewardship Fundamentals course by Dave Wells that I recently took.