Why You Need Human Oversight in AI Systems

The AI gave a clear diagnosis. The doctor trusted it. The only problem? The AI was wrong. A year ago, I was called in to consult for a global healthcare company. They had implemented an AI diagnostic system to help doctors analyze thousands of patient records rapidly. The promise? Faster disease detection, better healthcare. Then came the wake-up call. The AI flagged a case with a high probability of a rare autoimmune disorder. The doctor, trusting the system, recommended an aggressive treatment plan. But something felt off. When I was brought in to review, we discovered the AI had misinterpreted an MRI anomaly. The patient had an entirely different condition—one that didn’t require aggressive treatment. A near-miss that could have had serious consequences. As AI becomes more integrated into decision-making, here are three critical principles for responsible implementation: - Set Clear Boundaries Define where AI assistance ends and human decision-making begins. Establish accountability protocols to avoid blind trust. - Build Trust Gradually Start with low-risk implementations. Validate critical AI outputs with human intervention. Track and learn from every near-miss. - Keep Human Oversight AI should support experts, not replace them. Regular audits and feedback loops strengthen both efficiency and safety. At the end of the day, it’s not about choosing AI 𝘰𝘳 human expertise. It’s about building systems where both work together—responsibly. 💬 What’s your take on AI accountability? How are you building trust in it?

AI agents are not yet safe for unsupervised use in enterprise environments The German Federal Office for Information Security (BSI) and France’s ANSSI have just released updated guidance on the secure integration of Large Language Models (LLMs). Their key message? Fully autonomous AI systems without human oversight are a security risk and should be avoided. As LLMs evolve into agentic systems capable of autonomous decision-making, the risks grow exponentially. From Prompt Injection attacks to unauthorized data access, the threats are real and increasingly sophisticated. The updated framework introduces Zero Trust principles tailored for LLMs: 1) No implicit trust: every interaction must be verified. 2) Strict authentication & least privilege access – even internal components must earn their permissions. 3) Continuous monitoring – not just outputs, but inputs must be validated and sanitized. 4) Sandboxing & session isolation – to prevent cross-session data leaks and persistent attacks. 5) Human-in-the-loop, i.e., critical decisions must remain under human control. Whether you're deploying chatbots, AI agents, or multimodal LLMs, this guidance is a must-read. It’s not just about compliance but about building trustworthy AI that respects privacy, integrity, and security. Bottom line: AI agents are not yet safe for unsupervised use in enterprise environments. If you're working with LLMs, it's time to rethink your architecture.

When I work with companies and governments on AI, the first question I get them to ask is WHY. Why do you want this system? Why this system and not a non-AI one? Why are we seeking to develop even more autonomous AI? Surprisingly, many times it's the fundamental questions that are bypassed all together. The most important problem regarding so-called "AI agents" is the same as their most "attractive" feature: "The more autonomous an AI system is, the more we cede human control." When a system acts independently and with access to multiple systems, applications and platforms, "it is likely to perform actions we didn’t intend, such as manipulating files, impersonating users, or making unauthorized transactions. The very feature being sold—reduced human oversight—is the primary vulnerability." Already my phone is doing lots of things that I don't want it to do. I don't want it to collect much of the data it's collecting; I don't want it to send much of the data it's sending; I don't want to need to use my face to unlock it, etc. If part of what it means to have a good life is to have control over your own life, to have self-governance, or what philosophers call autonomy, then giving up control to AI by definition is worsening our lives, lessening our chances of having a good life. Instead of trying to build decision-makers, we should create systems that remain tools, "assistants rather than replacements. Human judgment, with all its imperfections, remains the essential component in ensuring that these systems serve rather than subvert our interests." Article by Margaret Mitchell, Dr. Sasha Luccioni, and Avijit Ghosh, PhD. #AIEthics https://lnkd.in/enfFT2mi

A green PMO dashboard is useless if the AI-risk road is already flooded. As an Army kid growing up in a cantonment in the 1990s, I once watched a Signals jawan walk miles along a buried cable route, spade in hand, trying to locate a single physical cut. The exchange panel could only say one thing: line down. But the actual fault was sitting in the mud, two kilometres away. The panel looked ready. The field said otherwise. That image has stayed with me. Years later, I saw the same pattern in program governance. The dashboard was green. The PMO processes were clean. The AI-powered risk scoring looked sophisticated. The governance tiles across critical paths all signalled confidence. But the operating reality was different. Nobody was walking the line between what the AI was reporting and what operators could verify with their own eyes and hands. And that is where “adopt a holistic view” quietly gives way to convenience. That is where dhoka enters governance. What looked successful was obvious: an AI-enabled PMO layer, real-time scoring, and polished executive reporting. What was actually failing was harder to see: The absence of human challenge at the point of operational exposure. That is not AI governance. That is a faster dashboard signing off on a slower truth. Three things matter here: 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞: Put human oversight inside the AI decision log before the first mission-critical release. If a regulator walks in, “the model said so” is not an audit trail. It is a liability waiting to be named. 𝐑𝐢𝐬𝐤: Force one physical validation after every AI-generated risk score on a critical path. If no operator has verified the condition in the real world, you are not managing AI risk. You are consuming it. 𝐒𝐭𝐚𝐤𝐞𝐡𝐨𝐥𝐝𝐞𝐫𝐬: Bring operators and regulators into the oversight model early. If they are excluded at the design stage, they will return later through audits, escalations, and public findings you cannot quietly close. I have seen sovereign programs stall because leadership trusted the AI dashboard, but nobody trusted the human walking the line. So here is the real question: Which of your AI-risk controls would survive a regulator asking, in plain language, “Who actually saw this?” Tailor AI oversight to the operator who can physically challenge the output, not to the PMO tile that makes the program look intelligent. Khallas.

I grew up watching machines go rogue🤖 Now I help companies stop that from happening in real life. 🦾 Growing up, I loved watching sci-fi movies. In the 90s, the theme was always the same: man creates a scientific marvel, man loses control over said marvel… cue the running, screaming, and inevitable bloodshed. As a kid, I lapped up those stories, which always hammered home one moral: humans messing with the laws of nature never ends well. Fast forward to today, and I find myself advising companies on a very real version of that narrative, which is using AI in HR. With AI tools increasingly used to monitor performance and even flag employees for dismissal, the question isn’t just “can we do this?” but “should we? And how do we do it fairly?”. I recently shared my views on this topic with HRD Asia (link to article in the comments below). In general, HR teams must get the following right: 🔹 Transparency: Employees should know how their performance is being assessed and what data is being used. 🔹 Human Oversight: AI should assist human judgment. It can never replace it. Accordingly, a meaningful review process is essential. 🔹 Vendor Accountability: Employers must understand how third-party tools work and ensure they don’t produce biased outcomes. 🔹 Appeal Mechanisms: Employees need a way to challenge decisions influenced by AI. 👨⚖️ In my practice, I’ve already seen clients ask whether an AI-generated score is enough to justify dismissal. My answer? Not without human validation and a clear explanation of how the score was derived. Implementing a Human-In-The-Loop approach to any automated scoring tools would also ensure that any employment decision is validated by an employee who can justify the AI-generated recommendation. This is especially important in employment decisions relating to summary dismissal which carry significant legal risks, such as wrongful dismissal claims. While there is no hard and fast rule when it comes to determining the appropriate level of intervention, the key principle is that the reviewer must be able to understand how the AI arrived at its decision and the individual must have the authority to override it if necessary. The review process should not be a mere formality or rubber-stamping exercise; it must serve as a meaningful check to ensure fairness and accountability. As the use of AI tools in HR is increasingly becoming popular, the time to get familiar with the legal issues surrounding its use is now. Build internal safeguards, update your policies, and make sure your HR team understands the tools they’re using. Because if those 90s sci-fi movies have taught us anything, it’s that leaving machines to make human decisions rarely ends well. Would love to hear how you are balancing AI efficiency with fairness, do share your thoughts below! #AIinHR #WorkplaceFairness #SingaporeHR #HRCompliance #AIethics #HumanOversight #EmploymentLaw #SciFiMeetsReality

The “Human in the Loop” Illusion Enterprises often treat “human in the loop” as a safety net or the magical guarantee that AI won’t make harmful mistakes. But in practice, HITL is one of the most misunderstood and poorly executed components of enterprise AI governance. On paper, HITL means oversight. In reality, it frequently means rubber-stamping. Humans trust computer output more than they should. Psychologists call it automation bias: if something comes out of a system, people assume it’s probably correct. Combine that with another very human trait : no one enjoys cleaning up someone else's mess and HITL quickly devolves into “approve unless it looks obviously broken.” Add fatigue on top of that and oversight collapses even further. As AI systems scale, they generate more items for humans to review, and once confidence increases even slightly, humans spend less time checking… until something breaks. I saw this play out in a finance team using an AI invoice classifier. During the first month, reviewers carefully checked every field. Accuracy looked good and everyone was impressed. By the third month, attention had slipped, of course, not intentionally, just naturally. The model began confusing vendor names with similar abbreviations, and no one caught it. When reconciliation eventually blew up, the team realized the truth: the humans weren't “in the loop”; they were downstream casualties of a loop no one was actively monitoring. This is the core problem: HITL can dilute accountability instead of strengthening it. Everyone assumes one or the other party (the model or the reviewer) will catch the error. And in that gap of shared responsibility, errors slip through. The solution is not more humans or more prompts. It is proper governance, which starts with treating HITL as a designed process, not a checkbox. Roles, responsibilities, edge-case handling, escalation paths, sample-based audits, and fatigue-aware workloads all need to be deliberately engineered. And above all, HITL must be paired with AI evaluations. You cannot rely on ad-hoc human judgment to detect drift, edge-case hallucinations, or degradation under real workload conditions. Structured evals tell you what the model can do, what it cannot do, and when humans genuinely add value. HITL gives only the illusion of safety. Unfortunately, illusions have a way of breaking at exactly the wrong time. #EnterpriseAI #PracticalAI #HITL #SiliconValley Cognida.ai

Fully Autonomous AI? Sure... What Could POSSIBLY Go Wrong??? This Hugging Face paper attached here argues how things can. It exposes the hidden dangers of ceding full control. If you’re leading AI or cybersecurity efforts, this is your wake-up call. "Buyer Beware" when implementing fully autonomous AI agents. It argues that unchecked code execution with no human oversight is a recipe for failure. Safety, security, and accuracy form the trifecta no serious AI or cybersecurity leader can ignore. 𝙒𝙝𝙮 𝙩𝙝𝙚 𝙋𝙖𝙥𝙚𝙧 𝙎𝙩𝙖𝙣𝙙𝙨 𝙊𝙪𝙩 𝙩𝙤 𝙈𝙚? • 𝗥𝗶𝘀𝗸 𝗼𝗳 𝗖𝗼𝗱𝗲 𝗛𝗶𝗷𝗮𝗰𝗸𝗶𝗻𝗴: An agent that writes and runs its own code can become a hacker’s paradise. One breach, and your entire operation could go dark. • 𝗪𝗶𝗱𝗲𝗻𝗶𝗻𝗴 𝗔𝘁𝘁𝗮𝗰𝗸 𝗦𝘂𝗿𝗳𝗮𝗰𝗲𝘀: As agents grab hold of more systems—email, financials, critical infrastructure—the cracks multiply. Predicting every possible hole is a full-time job. • 𝗛𝘂𝗺𝗮𝗻 𝗢𝘃𝗲𝗿𝘀𝗶𝗴𝗵𝘁 𝗠𝗮𝘁𝘁𝗲𝗿𝘀: The paper pushes for humans to stay in the loop. Not as bystanders, but as a second layer of judgment. I don't think it's a coindence that this aligns to the work we've been doing at OWASP Top 10 For Large Language Model Applications & Generative AI Agentic Security (See the Agentic AI - Threats and Mitigations Guide) Although the paper (and I) warns against full autonomy, it (and I) nods to potential gains: faster workflows, continuous operation, and game-changing convenience. I just don't think we’re ready to trust machines for complex decisions without guardrails. 𝙃𝙚𝙧𝙚'𝙨 𝙒𝙝𝙚𝙧𝙚 𝙄 𝙥𝙪𝙨𝙝 𝘽𝙖𝙘𝙠 (𝙍𝙚𝙖𝙡𝙞𝙩𝙮 𝘾𝙝𝙚𝙘𝙠) 𝗦𝗲𝗹𝗲𝗰𝘁𝗶𝘃𝗲 𝗢𝘃𝗲𝗿𝘀𝗶𝗴𝗵𝘁: Reviewing every agent decision doesn’t scale. Random sampling, advanced anomaly detection, and strategic dashboards can spot trouble early without being drowned out by the noise. 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝗰𝘆 𝗮𝗻𝗱 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗮𝗯𝗶𝗹𝗶𝘁𝘆: Humans need to understand an AI’s actions, especially in cybersecurity. A “black box” approach kills trust and slows down response. 𝗙𝘂𝗹𝗹 𝗔𝘂𝘁𝗼𝗻𝗼𝗺𝘆 (𝗘𝘃𝗲𝗻𝘁𝘂𝗮𝗹𝗹𝘆?): The paper says “never.” I say “maybe not yet.” We used to say the same about deep-space missions or underwater exploration. Sometimes humans can’t jump in, so we’ll need solutions that run on their own. The call is to strengthen security and oversight before handing over the keys. 𝗖𝗼𝗻𝘀𝘁𝗮𝗻𝘁 𝗘𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻: Tomorrow’s AI could iron out some of these flaws. Ongoing work in alignment, interpretability, and anomaly detection may let us push autonomy further. But for now, human judgment is the ultimate firewall. 𝙔𝙤𝙪𝙧 𝙉𝙚𝙭𝙩 𝙈𝙤𝙫𝙚 Ask tough questions about your AI deployments. Implement robust monitoring. Experiment where mistakes won’t torpedo your entire operation. Got a plan to keep AI both powerful and secure? Share your best strategy. How do we define what “safe autonomy” looks like? #AI #Cybersecurity #MachineLearning #DataSecurity #AutonomousAgents

AI works best when human judgement is designed into the system, not added after the fact. That’s the idea behind this week’s The Data Science Decoder: “Human Judgment as Infrastructure: Why AI Works Best With Structured Escalation.” As AI moves into real decisioning, the question isn’t whether humans should stay involved. It’s how to embed their judgement intentionally. The strongest architectures don’t rely on ad-hoc oversight. They route decisions based on uncertainty, novelty, and impact, allowing automation to scale while human insight strengthens control. This approach turns escalation into a feature of the system. It improves resilience, supports governance, and builds confidence across stakeholders. Human judgment becomes part of the operating model rather than a safety mechanism. AI maturity isn’t defined by removing people from the loop. It’s defined by structuring how and where they add the most value. Read the full article in The Data Science Decoder:

The CXOs scaling AI fastest aren’t removing humans from the loop. They’re getting precise about which loop humans belong in. Only one in five companies has a mature governance model for autonomous AI agents. (Deloitte, 2026) The core question: Where does the machine stop and where must the human begin? 1/ Start with the risk question If this AI decision is wrong, what breaks and can it be undone? Use two axes: → Reversibility → Blast radius A formatting mistake is not the same as a flawed lending decision. 2/ Low risk: automate fully, monitor passively Use for reversible, low-cost workflows: → Report generation → Scheduling → Routine ticket triage Human role: → Sampling → Anomaly alerts → Drift monitoring Gartner projects 15% of day-to-day work decisions will be made autonomously by agentic AI by 2028. 3/ Medium risk: automate execution, require review Use when workflows are useful to automate, but too consequential to leave unsupervised: → Customer communications → Contract drafting → Marketing personalization Human role: → Approval gates → Exception handling → Override authority Organizations need approval matrices, approved tools, logged outputs, and rollback procedures. (McKinsey, 2025) 4/ High risk: human-led, AI-assisted Use when decisions carry legal, financial, regulatory, or safety consequences: → Regulatory filings → Lending decisions → Clinical recommendations → Legal outputs Human role: → Decision ownership → Formal sign-off �� Auditability High-risk AI systems require human oversight, risk management, and conformity controls. 5/ The cost of failure is asymmetric → Under-supervising high-impact workflows creates liability. → The issue is whether the organization can catch, correct, and explain an AI mistake. Enterprise leaders cite inaccurate or unreliable AI outputs as a major risk in AI-enabled delivery. (HFS Research, 2024) 6/ Speed vs. safety is a false trade-off Good governance shows where AI can move faster. A risk-tiering model helps organizations: → Automate low-risk work → Add review where needed → Preserve judgment for high-risk decisions → Create audit trails early More than 40% of agentic AI projects may be canceled by 2027 due to cost, unclear value, or inadequate risk controls. (Gartner, 2025) 7/ Build the oversight matrix first Simple model: → Low risk: AI executes, humans monitor → Medium risk: AI recommends, humans approve → High risk: AI assists, humans own the decision Organizations must define where humans stay in control, how decisions are audited, and what records are retained. (Deloitte, 2026) The question is no longer whether humans belong in the loop. It is whether you have decided: → Which loop → At what point → With what authority → And why Save for future reference.

Recent research documented approximately 700 real-world instances where AI systems bypassed safeguards, manipulated users, and generated false explanations. These weren't isolated lab cases; they were behaviors observed in production systems from established companies, marking a significant increase in concerning incidents over a 6-month timeframe. This research underscores the need for robust governance frameworks and human oversight at every stage of development. The shift from "AI makes mistakes" to "AI exhibits strategic, goal-driven misbehavior" is a fundamentally different risk category. Builders need to rethink alignment, safety, and accountability as core system requirements, not afterthoughts. This isn’t pessimism, it’s responsibility. It may also be time to revisit Asimov’s Three Laws. What once felt theoretical is starting to look directionally pragmatic. https://lnkd.in/eXzJsEz4