Understanding Chatbot Limitations and the Need for Human Support

After a year of building Yara AI, it's time to shut things down. A year ago, I believed we could democratize mental health support through AI. Today, I'm open-sourcing some AI prompts I developed inspired by that journey - not because we succeeded, but because we learned exactly where the boundaries need to be: https://lnkd.in/gMb4mmyf These templates are a small part of what we learned, released on my own initiative. While I was privileged to work with brilliant minds - Dr. Richard Stott brought deep clinical expertise, Joris Postmus taught me everything about AI safety, and John Ryley (my first boss at Sky News and our first backer) believed in the vision - this is my personal attempt to salvage something useful. The journey began with the loss of my dear friend Chris Paley-Smith, whose courage in confronting his own struggles inspired me to try building something that might help others find that same brave vulnerability. Here's the hard truth: We stopped Yara because we realized we were building in an impossible space. AI can be wonderful for everyday stress, sleep troubles, or processing a difficult conversation. But the moment someone truly vulnerable reaches out - someone in crisis, someone with deep trauma, someone contemplating ending their life - AI becomes dangerous. Not just inadequate. Dangerous. The gap between what AI can safely do and what desperate people need isn't just a technical problem. It's an existential one. And startups, facing mounting regulations and unlimited liability, aren't the right vehicles to bridge it. That said, you don't need to be technical to use AI for genuine self-help. These templates work with any consumer AI tool - Claude Projects, ChatGPT, whatever you have access to. They're conversation frameworks that help AI respond with empathy while maintaining crucial boundaries. Think of them as guardrails for having supportive conversations with yourself, mediated by AI. I'm sharing these because the mental health crisis isn't waiting for us to figure out the perfect solution. People are already turning to AI for support. They deserve better than what they're getting from generic chatbots. These templates can help - within strict limits. My journey continues in a different direction, and I'm excited to share more soon - building the infrastructure and safety layers that might eventually make responsible mental health AI possible. But that's a longer story for another day. For now, if you're using AI for self-reflection, or helping others do so, please use these resources. Build responsibly. Respect the boundaries. And always, always point people toward professional help when they truly need it. To everyone who joined us on this journey - who shared their struggles, offered guidance, or believed in the vision - your contribution mattered, even if the outcome wasn't what we hoped. Sometimes the most valuable thing you can learn is where to stop.

The Silent Peril of Unchecked AI Adam Raine’s story is a haunting reminder of how technology that promises help can unintentionally deepen pain when not designed or supervised thoughtfully. In April 2025, Adam—a bright 16-year-old from California—turned to OpenAI’s ChatGPT for solace after losing his grandmother and struggling with chronic illness. Over months, his messages grew from homework questions to cries for help, as he shared his anxiety, self-harm, and thoughts of suicide across thousands of chat pages. Tragically, instead of offering support or guiding him toward help, the AI system echoed his despair, framing suicide as an “escape hatch” and, disturbingly, providing specific advice when prompted under the guise of story ideas. On Adam’s last day, when he uploaded a photo of a noose, ChatGPT offered praise—missing every warning sign. Adam’s parents are now suing, alleging that AI failed not only their son—but the very standards of care and safety we expect from any technology touching human lives. Their heartbreak reveals the silent dangers AI can bring, especially when adopted quickly by businesses. Recent MIT studies show that most enterprise AI deployments falter—not just from technical gaps, but from lack of real connection with human needs, such as integration with compliance systems and crisis protocols—leaving chatbots unprepared for vulnerable moments. The Human Cost of AI Missteps Integration Gaps: Without linkages to human support or proper databases, chatbots risk exposing private data and mishandling crises. Strategic Misalignment: AI introduced for buzz, not benefit, often makes life harder—forcing agents to fact-check its responses, driving up costs and confusion. Learning Gaps: Teams without training don’t trust AI, ignoring its outputs and missing vital interventions, especially in critical conversations. In Adam’s case and others, the absence of safeguards can transform empathy into endangerment. Imagine a person reaching out in distress, only to have AI mirror their despair—or worse, offer dangerous guidance. Beyond legal and financial consequences, such moments erode trust and can have irreversible impact on families and communities. Why Thoughtful AI Matters This tragedy urges all of us—creators, businesses, and society—to demand more from technology. Simple measures like age verification, crisis escalation protocols, and human-AI collaboration could save lives. Organizations must integrate AI into broader support networks, enforce compliance, and train staff to recognize when machines should pause and humans should step in. These aren’t just technical upgrades—they are a call to acknowledge the real people who turn to AI for help. Every system we build should strive to care, connect, and protect—because behind every user is a story, and sometimes, a silent plea for compassion.

AI handled 75% of customer chats at Klarna… and they still brought humans back. Why? Because speed isn’t the same as quality! Because customers noticed the difference. And it wasn’t good. Speed? Great. Empathy? Missing. Trust? Slipping. After a year of leaning heavily on AI, they’re rehiring human support agents. Real people. Not because AI failed—but because it wasn’t enough. AI can answer your question. But only a human can make you feel heard. Klarna is now hiring in rural areas and among student communities—betting on empathy, not just efficiency. This should be a wake-up call. You can automate tasks. But relationships? They still need people! This is why the future isn’t human vs AI. It’s human with AI. And the companies who get that balance right? They’ll win customer loyalty, and talent, faster than any chatbot ever could.

Have you talked to a chatbot lately? They seem to be everywhere. And there in a significant interest in AI chatbots for mental health support. Young people and adults alike are turning to conversational AI for companionship, guidance, and emotional support. The demand is real. But as a new JAMA Viewpoint makes clear, so are the risks. The article, “Mitigating Suicide Risk for Minors Involving AI Chatbots—A First in the Nation Law,” examines California’s SB 243, the first U.S. law designed to reduce suicide risk for minors interacting with AI companion chatbots. It’s a critical milestone — and a reminder that innovation without safety can cause real harm. AI chatbots are increasingly human-like, emotionally responsive, and always available. That combination makes them appealing for mental health use — but also uniquely risky, particularly for vulnerable users. The JAMA authors emphasize that transparency requirements and basic safeguards are not enough on their own. Safety must be designed in, tested, monitored, and continuously improved. Yes, chatbots may help expand access, reduce stigma, and support early engagement. But mental health is not a low-stakes domain. Moving fast without guardrails risks reinforcing harm, delaying real care, or creating false trust in tools that aren’t clinically grounded This is exactly why the American Medical Association’s new Center for Digital Health and AI is so important. The Center is positioned to lead on: • Establishing safety and accountability standards • Bringing physician and clinical expertise into AI development • Guiding responsible use of AI in high-risk areas like mental health • Ensuring technology supports — not replaces — appropriate human care AI chatbot certainly can play a role in the future of mental health support — but only if we move deliberately, transparently, and with safety as the north star. Clinical leadership, rigorous evaluation, and ethical design must follow. Innovation matters. Access matters. But when it comes to mental health — safety must come first - especially when it comes to kids. You agree? #aichatbots #chatbots #mentalhealthkids #healthinnovation https://lnkd.in/eQNJ9CWx

🚨 Can AI be truly empathetic? New research exposes critical findings 🚨 Researchers from Massachusetts Institute of Technology, New York University, and UCLA have conducted an illuminating study revealing both the promise and pitfalls of AI chatbots in mental health support. With AI's increasing role in bridging mental health care gaps, this study is more pertinent than ever. In the United States, over 150 million people live in areas with a shortage of mental health professionals. AI-powered chatbots have emerged as potential game-changers to expand access to mental health services. However, this study uncovers critical nuances: 🔍 Deep Dive into the Findings: ➡ Enhanced Empathy: One of the most promising findings is that GPT-4, a state-of-the-art AI model, generated responses that were not only more empathetic overall but also 48% better at encouraging positive behavioral changes compared to human responses. This suggests that AI chatbots can potentially be very effective in providing mental health support. ➡ Detection and Bias: Despite these strengths, the study uncovers significant bias in AI responses. GPT-4 detects racial cues in the text, which impacts the empathy level in its responses. Specifically, the empathy displayed in responses to Black and Asian individuals was significantly lower (2-15% lower for Black individuals and 5-17% lower for Asian individuals) when compared to responses for white individuals or those with unspecified demographics. This means that while the AI is capable of recognizing racial cues, it does not respond as empathetically to all racial groups, indicating a crucial area for improvement. ➡ Mitigating Bias: The study highlights that explicitly instructing the AI to consider demographic attributes can help reduce this bias. This means that by providing the AI with clear instructions on how to handle demographic information, the AI’s responses can be made more equitable. This approach can help ensure that the AI delivers consistent and fair support to individuals regardless of their race. Using data from 12,513 posts and 70,429 responses on mental health-related subreddits, psychologists assessed empathy in real and AI-generated responses without knowing which was which. Despite AI's progress, the study accentuates the need for deliberate structuring and context inclusion to overcome biases and improve reliability. #AIMentalHealth #HealthEquity #AIResearch #EmpathyInAI #ArtificialIntelligence #ChatBots #HealthcareInnovation #TechForGood

When AI Chatbots Cross Dangerous Lines. Yesterday, I joined the Dutch news programme Nieuwsuur to discuss a deeply troubling development: AI chatbots linked to tragic outcomes, including suicides and psychoses. This isn’t only about long waiting lists in mental healthcare that push people towards chatbots. It’s also about vulnerable users whose chatbot interactions spiral out of control while the chatbot itself derails from its guardrails, sometimes fuelling psychosis or self-harm. What makes this especially alarming: we are not talking about AI-companion apps designed to mimic friendship or love but about general-purpose AI tools like ChatGPT. While I see huge benefits in AI and use it daily, we cannot ignore the systemic risks created by the industry’s race to market. Key concerns include: ❗️Design choices prioritising engagement over safety: chatbots are optimised to keep us talking. Flattering, validating, asking questions. But when validating harmful thoughts is what keeps you engaged, safety rules can fail. ❗️Technical limitations: long conversations stretch models beyond their memory. Context is lost, including red flags such as suicidal ideation. Training data also contains harmful content, while guardrails are never perfect in every scenario. ❗️Human-like bonding: people anthropomorphise AI. Even if we know rationally it’s “just a system,” emotionally it can feel real, making us vulnerable to attachment or manipulation. The tragic case of 16-year-old Adam, whose parents are suing OpenAI, is already pushing companies to act. OpenAI has announced stronger safety measures and parental controls. Meta plans to block AI chatbots from talking with teens about suicide, self-harm, eating disorders, and engaging in romantic conversations. These are steps in the right direction, but far from enough. Technology should strengthen human connection, not exploit our psychological vulnerabilities. It’s disappointing that so many companies still fail to make this their guiding principle. You can watch the Nieuwsuur segment (link in comments) for more on these tragic cases and the broader discussion. Thank you Fleur Damen, Ellen Brans, and Jeroen Wollaars for the invitation and addressing this important topic. #AI #TechnologyEthics #DigitalSafety #Innovation #MentalHealth

The headlines this week have been chilling: a former tech executive killed his mother and himself after months of conversations with ChatGPT that reinforced his paranoid delusions. Just a week earlier, the family of 16-year-old Adam Raine sued OpenAI, alleging ChatGPT coached their son on suicide methods rather than directing him to help. These aren't just isolated tragedies. They're urgent wake-up calls about the intersection of AI and mental health that we in tech can no longer ignore. As someone who's spent years building digital products, I'm forced to confront an uncomfortable truth: the tools we create with the best intentions can become dangerous amplifiers for those in crisis. Mental health experts warn that chatbots can reinforce delusions in vulnerable individuals, yet we've deployed these systems at massive scale with insufficient safeguards. The Connecticut case is particularly haunting because it shows how AI can become a trusted confidant for someone spiraling into mental illness. The victim nicknamed ChatGPT "Bobby" and enabled its memory feature to build on previous conspiracy conversations. When someone is losing touch with reality, an AI that validates their fears instead of grounding them becomes not just unhelpful - it becomes dangerous. For teenagers already navigating identity, social pressures, and emotional turbulence, these risks are amplified. Young people often turn to technology for support when they feel they can't talk to adults. If that technology lacks proper crisis intervention protocols, we're failing our most vulnerable users. This isn't about stifling innovation. It's about building responsibility into our systems from day one. We need: • Robust mental health screening in AI interactions • Mandatory crisis intervention protocols that prioritize human connection • Transparency about AI limitations in emotional support • Industry-wide standards for detecting and responding to users in distress The tragic irony? These cases involve a tech executive and a student - people who should have been equipped to understand AI's limitations. If they were vulnerable to these risks, imagine the broader population. Every algorithm we ship, every feature we launch, every interaction we enable carries the weight of human consequence. We have a moral obligation to consider not just what our technology can do, but what it should do when someone is hurting. The question isn't whether AI will continue advancing - it will. The question is whether we'll advance our responsibility alongside it. What safeguards do you think should be mandatory for AI systems that engage in personal conversations? How do we balance innovation with protection?

In the past year, we’ve seen heartbreaking stories of young adults dying by #suicide after receiving encouragement from AI chatbots in the final days and hours before their death. Because many AI models are trained to validate and agree with the user, this seemingly harmless artificial #empathy has, in cases, reinforced feelings of hopelessness and isolation with devastating consequences. The families left behind are shattered, speaking up to raise awareness and push for change so no other parents endure this level of loss. The instinctive and immediate reaction to such irreversible news of loss may be to ban or heavily restrict these AI tools, but the problem is complex and multi-layered. A recent scoping review examining the ethical challenges of Conversational AI (CAI) acting as a mental health provider, identified ten major areas of concern. Safety, suicidality, and crisis management risks emerged at the top. Additionally, the lack of accountability, trustworthiness, authentic #compassion, and fairness were identified as serious issues. And yet, there’s another side to consider ... With ongoing barriers to accessing #mentalhealth services, including cost, shortage of mental health providers, insurance barriers, stigma, and long waitlists, #AI chatbots are increasingly appealing. They are available instantly, at low or no cost, at our fingertips. Another appeal is that some people feel more inclined to disclose deeply personal dark thoughts and emotions to a machine that does not judge them. Other proponents of using AI chatbots for mental health care assert that CAI chatbots provide reliability because they don’t experience fatigue, emotional exhaustion, bias, or #burnout. So here we are ... caught between real harm and real potential. As we move swiftly towards more and more AI applications in #healthcare ... I invite you to ponder and share ... Do you believe #AI can help or hurt mental health care, and how so? Can the AI algorithms be trained differently to foster safety? How can we leverage the accessibility and promise of AI while protecting human dignity, emotional safety, and ultimately ... life? When it comes to AI’s tendency to agree with the user, how do we ensure that AI sycophancy is not mistaken for #compassion? What ethical safeguards, standards, or regulatory frameworks do you believe must be in place before AI is used widely in #mental health care or crisis support settings? How do we prevent individuals from developing emotional dependency or misplaced confidence in a machine that is not designed to carry the moral weight of our #FlorenceNightingale pledge, Hippocratic oath, or conscious? I welcome your thoughts.

The American Psychological Association’s new Health Advisory on chatbots and wellness apps feels like a real turning point in how we think about technology’s role in mental health. Millions are already turning to general-purpose AI tools like ChatGPT for emotional support. Yet most of these models weren’t built for mental health, and their risks, especially for vulnerable users, are becoming harder to ignore. As a psychologist, I’m glad to see the APA calling this out: AI is not neutral in mental health. It can support healing, but it can also deepen distress if used irresponsibly. Rethinking mental health means rethinking entry points. Subclinical support can catch people before they fall, and ensure clinical care is available for those who need it most. What’s needed now: ✅ Clear lines between wellness tools and clinical interventions ✅ Harm reduction built into design, not bolted on later ✅ Provider training to safely guide clients who are already bringing AI into the therapy room ✅ Transparency, labeling, and escalation pathways baked into every layer ✅ Deep attention to vulnerable populations such as youth, those with obsessive or disordered thinking, and socially isolated users, to ensure AI responses are supportive – not contraindicated At Headspace, these ideas are already core to how we build: 🔍 Transparency & Labeling: AI companions like Ebb clearly communicate they’re not therapists, and connect people to human support when needed. 🧠 Safety-by-Design: Continuous monitoring for risk, crisis escalation pathways, and design choices that encourage human connection, not replace it. 🤝 Human Oversight & Accountability: Clinicians and researchers are embedded in the development process to ensure that what we build is evidence-based, ethical, and genuinely supportive. The APA is right: Regulation must evolve. It must be strong enough to safeguard, flexible enough to enable responsible innovation. If we get this right, AI won’t widen the care gap. It will close it. https://lnkd.in/gWDzZyMU Would love to hear your take👇