Algorithmic design and user trust

Last week at an AI healthcare summit, a Fortune 500 CTO admitted something disturbing: "We spent $7M on an enterprise AI system that sits unused. Nobody trusts it." And this is not the first time I have come across such cases. Having built an AI healthcare company in 2018 (before most people had even heard of transformers), I've witnessed this pattern from both sides: as a builder and as an advisor. The reality is that trust is the real bottleneck to AI adoption (not capability). I learned this firsthand when deploying AI in highly regulated healthcare environments. I have watched brilliant technical teams optimize models to 99% accuracy while ignoring the fundamental human question: "Why should I believe what this system tells me?" This creates a fascinating paradox that affects both enterprises, as well as people like you and me, so we can effectively use AI today: Users want AI that works autonomously (requiring less human input) yet remains interpretable (providing more human understanding). This tension is precisely where UI design becomes the determining factor in market success. Take Anthropic's Claude, for example. Its computer use feature reveals reasoning steps anyone can follow. It changes the experience from "AI did something" to "AI did something, and here's why" – making YOU more powerful without requiring technical expertise. The business impact speaks for itself: their enterprise adoption reportedly doubled after adding this feature. The pattern repeats across every successful AI product I have analyzed. Adept's command-bar overlay shows actions in real-time as it navigates your screen. This "show your work" approach cut rework by 75%, according to their case studies. These are not random enterprise solutions. They demonstrate how AI can 10x YOUR productivity today when designed with human understanding in mind. They prove a fundamental truth about human psychology: Users tolerate occasional AI mistakes if they can see WHY the mistake happened. What they won't tolerate is blind faith. Here's what nobody tells you about designing UI for AI that people actually adopt: • Make reasoning visible without overwhelming. Surface the logic, not just the answer • Signal confidence levels honestly. Users trust systems more when they admit uncertainty • Build correction loops that let people fix AI mistakes in seconds, not minutes • Include preview modes so users can verify before committing This is the sweet spot. — The market is flooded with capable AI. The shortage is in trusted AI that ordinary people can leverage effectively. The real moat is designing interfaces that earn user trust by clearly explaining AI's reasoning without needing technical expertise. The companies that solve for trust through thoughtful UI design will define the next wave of AI. Follow me Nicola for more insights on AI and how you can use it to make your life 10x better without requiring technical expertise.

In AI UX, I think we need to be much more careful with how we talk about trust. Too often, trust is treated like something we should simply increase, almost like adoption, satisfaction, or engagement. But trust in AI is not one clean product metric. A user can trust a system because it performs well, because it feels familiar, because it explains itself clearly, because it reduces effort, because it confirms what they already believe, or because it gives them distance from responsibility. Those are very different psychological processes, and if we measure them with one question like “Do you trust this AI?”, we lose most of what matters. For UX researchers, the important distinction is between whether the AI is actually worthy of trust, whether the user perceives it as trustworthy, whether the user says they are willing to rely on it, and whether they actually follow its recommendation in practice. These things can move in different directions. Someone may say they do not trust AI but still accept its suggestion because it is convenient. Someone else may say the system is reliable but avoid using it when the stakes feel high. That means self-report alone is not enough, and usage data alone is not enough either. We need to study the gap between what users say, what they believe, what they do, and what the system can actually support. This also means AI trust is not only about accuracy. Performance matters, of course. Users need to know whether the system is competent, consistent, and reliable. But users also make moral judgments. Is the system fair? Is it honest about uncertainty? Could it harm someone? A system can be highly capable and still feel unsafe, manipulative, biased, or inappropriate for a specific context. That is especially important in domains like hiring, healthcare, education, finance, legal decision-making, and public services, A better UX research question is: do users trust this specific system, for this specific task, in this specific context, with this level of risk, and with these possible consequences? People may trust AI for navigation, grammar support, or summarizing a document, but hesitate when the same kind of technology is used for diagnosis, grading, hiring, or moral advice. Trust is always attached to an object, a situation, and a perceived risk. Treating “AI trust” as one general attitude makes our research too vague. I also think we need to pay more attention to motivated reliance. Sometimes people rely on AI not because it is more trustworthy, but because it is easier, faster, more convenient, or useful for justifying a decision they already wanted to make. In those cases, overreliance is not just a literacy problem. It can be a motivation problem. Users may want the AI to take on the cognitive burden, the uncertainty, or even part of the moral responsibility. That is a very different design challenge from simply making the system clearer.

As AI becomes integral to our daily lives, many still ask: can we trust its output? That trust gap can slow progress, preventing us from seeing AI as a tool. Transparency is the first step. When an AI system suggests an action, showing the key factors behind that suggestion helps users understand the “why” rather than the “what”. By revealing that a recommendation that comes from a spike in usage data or an emerging seasonal trend, you give users an intuitive way to gauge how the model makes its call. That clarity ultimately bolsters confidence and yields better outcomes. Keeping a human in the loop is equally important. Algorithms are great at sifting through massive datasets and highlighting patterns that would take a human weeks to spot, but only humans can apply nuance, ethical judgment, and real-world experience. Allowing users to review and adjust AI recommendations ensures that edge cases don’t fall through the cracks. Over time, confidence also grows through iterative feedback. Every time a user tweaks a suggested output, those human decisions retrain the model. As the AI learns from real-world edits, it aligns more closely with the user’s expectations and goals, gradually bolstering trust through repeated collaboration. Finally, well-defined guardrails help AI models stay focused on the user’s core priorities. A personal finance app might require extra user confirmation if an AI suggests transferring funds above a certain threshold, for example. Guardrails are about ensuring AI-driven insights remain tethered to real objectives and values. By combining transparent insights, human oversight, continuous feedback, and well-defined guardrails, we can transform AI from a black box into a trusted collaborator. As we move through 2025, the teams that master this balance won’t just see higher adoption: they’ll unlock new realms of efficiency and creativity. How are you building trust in your AI systems? I’d love to hear your experiences. #ArtificialIntelligence #RetailAI

Humanizing AI Through the Kano Model In an era where generative AI has become a ubiquitous offering, true differentiation lies not in merely adopting the technology but in integrating human values into its core. Building on my earlier discussion about applying the Kano Model to Gen AI strategy, let’s explore how this framework can refocus development metrics to prioritize ethics and human-centricity. By aligning AI systems with human needs, organizations can shift from functional tools to trusted partners that inspire lasting loyalty. Traditional metrics such as speed, scalability, and model accuracy have evolved into basic expectations the “must-haves” of AI. What truly elevates a product today is its ability to embody values like safety, helpfulness, dignity, and harmlessness. These qualities, categorized as “delighters” in the Kano Model, transform AI from a transactional tool into a meaningful collaborator. Key Human-Centric Differentiators Safety: Proactive safeguards must ensure AI systems protect users from risks, whether physical, emotional, or societal. Safety is non-negotiable in building trust. Helpfulness: Personalized, context-aware interactions demonstrate empathy. AI should anticipate needs and adapt to individual preferences, turning routine tasks into meaningful experiences. Dignity: Ethical design principles—fairness, transparency, and privacy—must underpin AI development. Respecting user autonomy fosters long-term trust and engagement. Harmlessness: AI outputs and recommendations should prioritize user well-being, avoiding unintended consequences like bias, misinformation, or psychological harm. This human-centered approach represents a paradigm shift in technology development. While traditional KPIs remain important, they are no longer sufficient to stand out in a crowded market. Organizations that embed human values into their AI systems will not only meet user expectations but exceed them, creating emotional connections that drive loyalty. By applying the Kano Model, businesses can systematically align innovation with ethics, ensuring technology serves humanity rather than the other way around. The future of AI isn’t just about efficiency it’s about elevating human potential through thoughtful, responsible design. How is your organization balancing technical excellence with human values?

Building trust rests on three pillars: authenticity, empathy, and logic (as articulated by Frances Frei in their TED talk). Humans learn to establish trust with one another through repeated interactions, testing boundaries, and lived experience. We are now entering that same trust-building journey with AI. When people use AI tools, the first implicit question is often: Can I trust this system with its answers or actions? That trust may initially be borrowed from the brand behind the tool, but recent progression shows that brand trust alone does not sustain confidence. Users will test systems for themselves. As Satya Nadella has mentioned, the technology industry has no lasting franchise value and trust must be earned continuously. Today, AI performs relatively well on two of the three pillars, at least on paper, in benchmarks, and often in individual user experiences. First, logic, while still debated in terms of true “reasoning”, has improved significantly. Second, Empathy, in some cases, appears surprisingly strong, even exceeding human expectations in tone and responsiveness. The missing pillar is authenticity. AI often struggles to demonstrate a grounded sense of truthfulness and conviction in its responses or actions. This is an uphill challenge for the technology, made harder by the fact that authenticity is difficult to define and even harder to measure. There are few, if any, robust metrics to assess it. Ironically, the pursuit of empathy can actively erode authenticity. In trying to be helpful and agreeable, AI systems often default to pleasing the user, even when the user is wrong. They rarely respond with confident disagreement or a firm “no.” The implicit assumption becomes that the user is always right, regardless of accuracy. Over time, this dynamic weakens trust rather than strengthening it. Authenticity in AI will ultimately depend on being willing to be honest, grounded, and occasionally uncomfortable. These are the qualities that humans instinctively associate with true trust. AGI will not be just about technology, but about trust in it. #ExperienceFromTheField #WrittenByHuman

One of the most common mistakes in AI system design is the attempt to eliminate uncertainty. Teams chase higher accuracy, tighter logic, cleaner prompts, assuming that with enough refinement a system can behave predictably in every situation. The impulse is understandable. It is also misplaced. Agentic AI systems operate in probabilities, not guarantees. No amount of optimization removes uncertainty entirely. Instead, we need to think about how the system should behave when uncertainty inevitably appears. Trustworthy systems are defined by restraint. They know when to pause, when to defer, and when to escalate. They are designed to recognize ambiguity and respond safely, rather than forcing a decision where one should not be made. Many systems fail for a simple reason. They are implicitly rewarded for producing an answer, not for producing the right behavior. When uncertainty is treated as failure, the system learns to conceal it. That is a design choice. Responsible design starts with clearly defining where autonomy ends. It means setting explicit thresholds for deferral, escalation, and human intervention. It means prioritizing correctness and safety over completeness. Paradoxically, accepting uncertainty increases reliability. A system that can acknowledge “I don’t know” can behave more predictably than one that must always respond. But how much of this acceptable to business users? The goal is bounded autonomy with accountability: AI systems execute actions, humans remain responsible for outcomes.

𝗖𝘂𝘀𝘁𝗼𝗺𝗲𝗿𝘀 𝗱𝗼𝗻'𝘁 𝘄𝗮𝗻𝘁 𝗺𝗮𝗴𝗶𝗰𝗮𝗹 𝗔𝗜. 𝗧𝗵𝗲𝘆 𝘄𝗮𝗻𝘁 𝗰𝗼𝗻𝘁𝗿𝗼𝗹. We talk a lot about the "Agentic Era", but the customers I speak with are hesitant to hand over the keys to their workflows. And I don’t blame them. We are asking them to trust black boxes with business-critical work. I wrote ✨ The Internet Principles of AI ✨ because we are at the exact same point the internet was in the early 90s. The raw power was there, but it wasn’t useful to a normal person until we built the harness: the browser, the protocols, and the safety layers. That harness wasn't for the machines. It was for the people. If we want to win real trust, AI needs that same spine. In the piece, I break down what that looks like practically: 🔹 Responsibility at the Edge. Customers deserve a plain-text explanation of why an agent made a decision. Every time. 🔹 Proof over Promise. Agents should operate on "Act, Ask, or Abstain" logic. No surprises. 🔹 Honest Layers. We owe it to users to build architecture that admits when it's wrong, rather than hallucinating an answer. As we build the next frontier of AI, it's all about respecting the customer by giving them a system they can audit, understand, and control. Read the full piece below. ⬇️ #ProductLeadership #CustomerTrust #AgenticAI

*To build trust in complexity, offer small choices and fast feedback* I strongly believe product simplicity and predictability are a superpower. They give the user a sense of control, which is a gift when the world feels so complicated. But some things are legitimately complex. What gives the user a sense of control when predictability is hard to come by? My take: Giving the user a chance to *participate* in the process by laying out steps, enabling them to make specific choices, and offering a clear feedback loop on each small decision. This may make the flow longer, but it gives users a chance to viscerally understand what’s happening. A while ago I got an alarming privacy notification on an important account. I was immediately worried. But the product’s recovery flow calmed me down. Why? It: 1. Laid out all the steps I’d go through, giving me a clear roadmap for what to do. 2. Channeled my anxiety into actions, even if they were small. There were prompts like “Check whether password is compromised? Yes / No”. Is that a necessary prompt? Who would say “no”? But in the moment, the ability to participate in the process of securing my account gave me a sense of control. 3. Gave me fast feedback on each choice by turning each step green on completion. By the end of the list, I felt a sense of relief. Realistically, that product could have taken all those actions without my input. But getting to participate in each step gave me a sense of control. I saw the same thing with a new AI tool my team was working on. Our temptation was to take user input up front and come back with a solution. But our customers didn’t yet trust the magic black box of AI recommendations. Instead, what helped was inserting feedback steps explaining what we were considering and offering the user a chance to change direction at each step. It added friction, but it built trust faster. Then over time, we could remove those interim feedback steps and automatically make decisions. Compare that to a customer service page where you type a question into a contact form and get a message that says, “Thanks, we’ll take care of it.” You don’t really get an understanding of the overall process, a chance to make smaller decisions, or feedback on whether you made the right choices. I’m always stressed about whether I did it right! This applies to people too. When I’m building a relationship with a new manager or peer, I try to frequently outline what I’m doing and give them a chance to redirect. After a few weeks, we know each others’ style and I can stop. Action is the best antidote to fear. Especially when someone is stressed out and longing for control, it helps to ground them in a clear step-by-step process, give them a chance to participate in solving their problem, and letting them know the impact of each choice. That naturally creates some relief, and helps them channel their concern into action. (For regular updates, check out amivora.substack.com!)