Risks of Uncontrolled AI Infrastructure

"Autonomous AI agents—goal-directed, intelligent systems that can plan tasks, use external tools, and act for hours or days with minimal guidance—are moving from research labs into mainstream operations. But the same capabilities that drive efficiency also open new fault lines. An agent that can stealthily obtain and spend millions of dollars, cripple a main power line, or manipulate critical infrastructure systems would be disastrous. This report identifies three pressing risks from AI agents. First, catastrophic misuse: the same capabilities that streamline business could enable cyber-intrusions or lower barriers to dangerous attacks. Second, gradual human disempowerment: as more decisions migrate to opaque algorithms, power drifts away from human oversight long before any dramatic failure occurs. Third, workforce displacement: decision-level automation spreads faster and reaches deeper than earlier software waves, putting both employment and wage stability under pressure. Goldman Sachs projects that tasks equivalent to roughly 300 million full-time positions worldwide could be automated. In light of these risks, Congress should: 1. Create an Autonomy Passport. Before releasing AI agents with advanced capabilities such as handling money, controlling devices, or running code, companies should register them in a federal system that tracks what the agent can do, where it can operate, how it was tested for safety, and who to contact in emergencies. 2. Mandate continuous oversight and recall authority. High-capability agents should operate within digital guardrails that limit them to pre-approved actions, while CISA maintains authority to quickly suspend problematic deployments when issues arise. 3. Keep humans in the loop for high consequence domains. When an agent recommends actions that could endanger life, move large sums, or alter critical infrastructure, a professional, e.g., physician, compliance officer, grid engineer, or authorized official, must review and approve the action before it executes. 4. Monitor workforce impacts. Direct federal agencies to publish annual reports tracking job displacement and wage trends, building on existing bipartisan proposals like the Jobs of the Future Act to provide ready-made legislative language. These measures are focused squarely on where autonomy creates the highest risk, ensuring that low-risk innovation can flourish. Together, they act to protect the public and preserve American leadership in AI before the next generation of agents goes live. Good work from Joe K. at the Center for AI Policy

Concerned about agentic AI risks cascading through your system? Consider these emerging smart practices which adapt existing AI governance best practices for agentic AI, reinforcing a "responsible by design" approach and encompassing the AI lifecycle end-to-end: ✅ Clearly define and audit the scope, robustness, goals, performance, and security of each agent's actions and decision-making authority. ✅ Develop "AI stress tests" and assess the resilience of interconnected AI systems ✅ Implement "circuit breakers" (a.k.a kill switches or fail-safes) that can isolate failing models and prevent contagion, limiting the impact of individual AI agent failures. ✅ Implement human oversight and observability across the system, not necessarily requiring a human-in-the-loop for each agent or decision (caveat: take a risk-based, use-case dependent approach here!). ✅ Test new agents in isolated / sand-box environments that mimic real-world interactions before productionizing ✅ Ensure teams responsible for different agents share knowledge about potential risks, understand who is responsible for interventions and controls, and document who is accountable for fixes. ✅ Implement real-time monitoring and anomaly detection to track KPIs, anomalies, errors, and deviations to trigger alerts.

This new guide from the OWASP® Foundation Agentic Security Initiative for developers, architects, security professionals, and platform engineers building or securing agentic AI applications, published Feb 17, 2025, provides a threat-model-based reference for understanding emerging agentic AI threats and their mitigations. Link: https://lnkd.in/gFVHb2BF * * * The OWASP Agentic AI Threat Model highlights 15 major threats in AI-driven agents and potential mitigations: 1️⃣ Memory Poisoning – Prevent unauthorized data manipulation via session isolation & anomaly detection. 2️⃣ Tool Misuse – Enforce strict tool access controls & execution monitoring to prevent unauthorized actions. 3️⃣ Privilege Compromise – Use granular permission controls & role validation to prevent privilege escalation. 4️⃣ Resource Overload – Implement rate limiting & adaptive scaling to mitigate system failures. 5️⃣ Cascading Hallucinations – Deploy multi-source validation & output monitoring to reduce misinformation spread. 6️⃣ Intent Breaking & Goal Manipulation – Use goal alignment audits & AI behavioral tracking to prevent agent deviation. 7️⃣ Misaligned & Deceptive Behaviors – Require human confirmation & deception detection for high-risk AI decisions. 8️⃣ Repudiation & Untraceability – Ensure cryptographic logging & real-time monitoring for accountability. 9️⃣ Identity Spoofing & Impersonation – Strengthen identity validation & trust boundaries to prevent fraud. 🔟 Overwhelming Human Oversight – Introduce adaptive AI-human interaction thresholds to prevent decision fatigue. 1️⃣1️⃣ Unexpected Code Execution (RCE) – Sandbox execution & monitor AI-generated scripts for unauthorized actions. 1️⃣2️⃣ Agent Communication Poisoning – Secure agent-to-agent interactions with cryptographic authentication. 1️⃣3️⃣ Rogue Agents in Multi-Agent Systems – Monitor for unauthorized agent activities & enforce policy constraints. 1️⃣4️⃣ Human Attacks on Multi-Agent Systems – Restrict agent delegation & enforce inter-agent authentication. 1️⃣5️⃣ Human Manipulation – Implement response validation & content filtering to detect manipulated AI outputs. * * * The Agentic Threats Taxonomy Navigator then provides a structured approach to identifying and assessing agentic AI security risks by leading though 6 questions: 1️⃣ Autonomy & Reasoning Risks – Does the AI autonomously decide steps to achieve goals? 2️⃣ Memory-Based Threats – Does the AI rely on stored memory for decision-making? 3️⃣ Tool & Execution Threats – Does the AI use tools, system commands, or external integrations? 4️⃣ Authentication & Spoofing Risks – Does AI require authentication for users, tools, or services? 5️⃣ Human-In-The-Loop (HITL) Exploits – Does AI require human engagement for decisions? 6️⃣ Multi-Agent System Risks – Does the AI system rely on multiple interacting agents?

In his upcoming book, AI: Unexplainable, Unpredictable, Uncontrollable, Dr. Roman V. Yampolskiy, an AI safety expert, argues that there is currently no evidence to suggest that artificial intelligence (AI), particularly superintelligent systems, can be safely controlled. He emphasizes that the AI control problem is poorly understood and under-researched, despite its critical importance to humanity’s future. Dr. Yampolskiy’s extensive review of AI literature reveals that advanced intelligent systems possess the ability to learn new behaviors, adjust performance, and operate semi-autonomously in novel situations. This adaptability makes them inherently unpredictable and uncontrollable. He points out that as AI systems become more capable, their autonomy increases while human control diminishes, leading to potential safety risks. One significant challenge is that superintelligent AI can make decisions and encounter failures in an infinite number of ways, making it impossible to predict and mitigate all potential safety issues. Additionally, these systems often operate as “black boxes,” providing decisions without understandable explanations, which complicates efforts to ensure their safety and alignment with human values. Dr. Yampolskiy argues that the assumption of solvability in controlling AI is unfounded, as there is no proof supporting this belief. He suggests that the AI community should focus on minimizing risks while maximizing potential benefits, acknowledging that advanced intelligent systems will always present some level of risk. He also proposes that society must decide between relinquishing control to potentially beneficial but autonomous AI systems or maintaining human control at the expense of certain capabilities. In conclusion, Dr. Yampolskiy calls for increased efforts and funding for AI safety and security research, emphasizing the need to use this opportunity wisely to make AI systems as safe as possible, even if complete safety cannot be guaranteed. #technology

A lot of companies think they’re “safe” from AI compliance risks simply because they haven’t formally adopted AI. But that’s a dangerous assumption—and it’s already backfiring for some organizations. Here’s what’s really happening— Employees are quietly using ChatGPT, Claude, Gemini, and other tools to summarize customer data, rewrite client emails, or draft policy documents. In some cases, they’re even uploading sensitive files or legal content to get a “better” response. The organization may not have visibility into any of it. This is what’s called Shadow AI—unauthorized or unsanctioned use of AI tools by employees. Now, here’s what a #GRC professional needs to do about it: 1. Start with Discovery: Use internal surveys, browser activity logs (if available), or device-level monitoring to identify which teams are already using AI tools and for what purposes. No blame—just visibility. 2. Risk Categorization: Document the type of data being processed and match it to its sensitivity. Are they uploading PII? Legal content? Proprietary product info? If so, flag it. 3. Policy Design or Update: Draft an internal AI Use Policy. It doesn’t need to ban tools outright—but it should define: • What tools are approved • What types of data are prohibited • What employees need to do to request new tools 4. Communicate and Train: Employees need to understand not just what they can’t do, but why. Use plain examples to show how uploading files to a public AI model could violate privacy law, leak IP, or introduce bias into decisions. 5. Monitor and Adjust: Once you’ve rolled out your first version of the policy, revisit it every 60–90 days. This field is moving fast—and so should your governance. This can happen anywhere: in education, real estate, logistics, fintech, or nonprofits. You don’t need a team of AI engineers to start building good governance. You just need visibility, structure, and accountability. Let’s stop thinking of AI risk as something “only tech companies” deal with. Shadow AI is already in your workplace—you just haven’t looked yet.

AI is not failing because of bad ideas; it’s "failing" at enterprise scale because of two big gaps: 👉 Workforce Preparation 👉 Data Security for AI While I speak globally on both topics in depth, today I want to educate us on what it takes to secure data for AI—because 70–82% of AI projects pause or get cancelled at POC/MVP stage (source: #Gartner, #MIT). Why? One of the biggest reasons is a lack of readiness at the data layer. So let’s make it simple - there are 7 phases to securing data for AI—and each phase has direct business risk if ignored. 🔹 Phase 1: Data Sourcing Security - Validating the origin, ownership, and licensing rights of all ingested data. Why It Matters: You can’t build scalable AI with data you don’t own or can’t trace. 🔹 Phase 2: Data Infrastructure Security - Ensuring data warehouses, lakes, and pipelines that support your AI models are hardened and access-controlled. Why It Matters: Unsecured data environments are easy targets for bad actors making you exposed to data breaches, IP theft, and model poisoning. 🔹 Phase 3: Data In-Transit Security - Protecting data as it moves across internal or external systems, especially between cloud, APIs, and vendors. Why It Matters: Intercepted training data = compromised models. Think of it as shipping cash across town in an armored truck—or on a bicycle—your choice. 🔹 Phase 4: API Security for Foundational Models - Safeguarding the APIs you use to connect with LLMs and third-party GenAI platforms (OpenAI, Anthropic, etc.). Why It Matters: Unmonitored API calls can leak sensitive data into public models or expose internal IP. This isn’t just tech debt. It’s reputational and regulatory risk. 🔹 Phase 5: Foundational Model Protection - Defending your proprietary models and fine-tunes from external inference, theft, or malicious querying. Why It Matters: Prompt injection attacks are real. And your enterprise-trained model? It’s a business asset. You lock your office at night—do the same with your models. 🔹 Phase 6: Incident Response for AI Data Breaches - Having predefined protocols for breaches, hallucinations, or AI-generated harm—who’s notified, who investigates, how damage is mitigated. Why It Matters: AI-related incidents are happening. Legal needs response plans. Cyber needs escalation tiers. 🔹 Phase 7: CI/CD for Models (with Security Hooks) - Continuous integration and delivery pipelines for models, embedded with testing, governance, and version-control protocols. Why It Matter: Shipping models like software means risk comes faster—and so must detection. Governance must be baked into every deployment sprint. Want your AI strategy to succeed past MVP? Focus and lock down the data. #AI #DataSecurity #AILeadership #Cybersecurity #FutureOfWork #ResponsibleAI #SolRashidi #Data #Leadership

Unmasking AI Excerpt published by MIT Technology Review -“The term ‘x-risk is used as a shorthand for the hypothetical existential risk posed by AI. While my research supports the idea that AI systems should not be integrated into weapons systems because of the lethal dangers, this isn’t because I believe AI systems by themselves pose an existential risk as superintelligent agents. … When I think of x-risk, I think of the people being harmed now and those who are at risk of harm from AI systems. I think about the risk and reality of being “excoded.” You can be excoded when a hospital uses AI for triage and leaves you without care, or uses a clinical algorithm that precludes you from receiving a life-saving organ transplant. You can be excoded when you are denied a loan based on algorithmic decision-making. You can be excoded when your résumé is automatically screened out and you are denied the opportunity to compete for the remaining jobs that are not replaced by AI systems. You can be excoded when a tenant-screening algorithm denies you access to housing. All of these examples are real. No one is immune from being excoded, and those already marginalized are at greater risk… Though it is tempting to view physical violence as the ultimate harm, doing so makes it easy to forget pernicious ways our societies perpetuate structural violence. The Norwegian sociologist Johan Galtung coined this term to describe how institutions and social structures prevent people from meeting their fundamental needs and thus cause harm. Denial of access to health care, housing, and employment through the use of AI perpetuates individual harms and generational scars. AI systems can kill us slowly.” Read more in the full #UnmaskingAI book available today in print and via audiobook. www.unmasking.ai https://lnkd.in/efdByggM

This is yet another reason why you need a Secure AI solution if you're exploring anything AI related. Research has uncovered a vulnerability in Microsoft 365 Copilot that allowed hackers to access sensitive information without any user interaction. This “zero-click” flaw, dubbed EchoLeak, could have exposed confidential data from emails, spreadsheets, and chats with nothing more than a cleverly crafted email quietly read by the AI assistant. Executive Summary - Security researchers at Aim Security discovered that Microsoft 365 Copilot was susceptible to a novel form of attack: hackers could send an email containing hidden instructions, which Copilot would process automatically, leading to unauthorized access and sharing of internal data. No phishing links or malware were needed—just the AI’s own background scanning was enough to trigger the breach. - The vulnerability wasn’t just a minor bug; it revealed a fundamental design weakness in how AI agents handle trusted and untrusted data. This mirrors the early days of software security, when attackers first learned to hijack devices through overlooked flaws. Microsoft has since patched the issue and implemented additional safeguards, but the episode raises broader concerns about the security of all AI-powered agents. - The real risk isn’t limited to Copilot. Similar AI agents across the industry, from customer service bots to workflow assistants, could be vulnerable to the same kind of manipulation. The challenge lies in the unpredictable nature of AI and the vast attack surface that comes with integrating these agents into critical business processes. My Perspective As organizations race to harness the productivity gains of AI, this incident serves as a stark reminder: innovation must go hand-in-hand with robust security. The EchoLeak vulnerability highlights how AI’s ability to autonomously process instructions can become a double-edged sword—especially when the line between trusted and untrusted data is blurred. Until AI agents can reliably distinguish between legitimate commands and malicious prompts, every new integration is a potential risk. The Future Looking ahead, expect to see a surge in research and investment focused on fundamentally redesigning how AI agents interpret and act on information. For now, widespread adoption of autonomous AI agents in sensitive environments will remain cautious, as organizations grapple with these emerging threats. What You Should Think About If you’re deploying or experimenting with AI agents, now is the time to audit your systems, ask tough questions about how data and instructions are handled, and push vendors for transparency on security measures. Share your experiences or concerns: How are you balancing innovation with risk in your AI projects? What additional safeguards would you like to see? Let’s keep this conversation going and help shape a safer future for AI in the enterprise. Source: fortune

One of the most important contributions of Google DeepMind's new AGI Safety and Security paper is a clean, actionable framing of risk types. Instead of lumping all AI risks into one “doomer” narrative, they break it down into 4 clear categories- with very different implications for mitigation: 1. Misuse → The user is the adversary This isn’t the model behaving badly on its own. It’s humans intentionally instructing it to cause harm- think jailbreak prompts, bioengineering recipes, or social engineering scripts. If we don’t build strong guardrails around access, it doesn’t matter how aligned your model is. Safety = security + control 2. Misalignment → The AI is the adversary The model understands the developer’s intent- but still chooses a path that’s misaligned. It optimizes the reward signal, not the goal behind it. This is the classic “paperclip maximizer” problem, but much more subtle in practice. Alignment isn’t a static checkbox. We need continuous oversight, better interpretability, and ways to build confidence that a system is truly doing what we intend- even as it grows more capable. 3. Mistakes → The world is the adversary Sometimes the AI just… gets it wrong. Not because it’s malicious, but because it lacks the context, or generalizes poorly. This is where brittleness shows up- especially in real-world domains like healthcare, education, or policy. Don’t just test your model- stress test it. Mistakes come from gaps in our data, assumptions, and feedback loops. It's important to build with humility and audit aggressively. 4. Structural Risks → The system is the adversary These are emergent harms- misinformation ecosystems, feedback loops, market failures- that don’t come from one bad actor or one bad model, but from the way everything interacts. These are the hardest problems- and the most underfunded. We need researchers, policymakers, and industry working together to design incentive-aligned ecosystems for AI. The brilliance of this framework: It gives us language to ask better questions. Not just “is this AI safe?” But: - Safe from whom? - In what context? - Over what time horizon? We don’t need to agree on timelines for AGI to agree that risk literacy like this is step one. I’ll be sharing more breakdowns from the paper soon- this is one of the most pragmatic blueprints I’ve seen so far. 🔗Link to the paper in comments. -------- If you found this insightful, do share it with your network ♻️ Follow me (Aishwarya Srinivasan) for more AI news, insights, and educational content to keep you informed in this hyperfast AI landscape 💙