Key Risks in AI Development

📢 What are the risks from Artificial Intelligence? We present the AI Risk Repository: a comprehensive living database of 700+ risks extracted, with quotes and page numbers, from 43(!) taxonomies. To categorize the identified risks, we adapt two existing frameworks into taxonomies. Our Causal Taxonomy categorizes risks based on three factors: the Entity involved, the Intent behind the risk, and the Timing of its occurrence. Our Domain Taxonomy categorizes AI risks into 7 broad domains and 23 more specific subdomains. For example, 'Misinformation' is one of the domains, while 'False or misleading information' is one of its subdomains. 💡 Four insights from our analysis: 1️⃣ 51% of the risks extracted were attributed to AI systems, while 34% were attributed to humans. Slightly more risks were presented as being unintentional (37%) than intentional (35%). Six times more risks were presented as occurring after (65%) than before deployment (10%). 2️⃣ Existing risk frameworks vary widely in scope. On average, each framework addresses only 34% of the risk subdomains we identified. The most comprehensive framework covers 70% of these subdomains. However, nearly a quarter of the frameworks cover less than 20% of the subdomains. 3️⃣ Several subdomains, such as *Unfair discrimination and misrepresentation* (mentioned in 63% of documents); *Compromise of privacy* (61%); and *Cyberattacks, weapon development or use, and mass harm* (54%) are frequently discussed. 4️⃣ Others such as *AI welfare and rights* (2%), *Competitive dynamics* (12%), and *Pollution of information ecosystem and loss of consensus reality* (12%) were rarely discussed. 🔗 How can you engage?   Visit our website, explore the repository, read our preprint, offer feedback, or suggest missing resources or risks (see links in comments). 🙏 Please help us spread the word by sharing this with anyone relevant. Thanks to everyone involved: Alexander Saeri, Jess Graham 🔸, Emily Grundy, Michael Noetel 🔸, Risto Uuk, Soroush J. Pour, James Dao, Stephen Casper, and Neil Thompson. #AI #technology

✨ AI at a crossroads: Can we steer it responsibly? The Association for the Advancement of Artificial Intelligence (AAAI) 2025 Presidential Panel on the Future of AI Research lays out a stark reality—AI is advancing at an unprecedented pace, but governance, safety, and evaluation mechanisms are struggling to keep up. 🌏 Having worked at the intersection of AI governance, responsible deployment, and multi-agent AI, I see a recurring challenge: we are building AI that is more powerful than our ability to govern it responsibly. 🔬 Key takeaways from the report & my perspective:- ✅ AI Reasoning & Trustworthiness:- While LLMs and Agentic AI are demonstrating emergent reasoning, we lack verifiable correctness. Can we afford AI-driven decision-making without reliability guarantees? ✅ Agentic AI & Multi-Agent Systems:- The integration of LLMs into autonomous, multi-agent AI systems is a double-edged sword. On one hand, these systems offer adaptive, cooperative intelligence—but on the other, they introduce complexity, opacity, and safety risks. We need governance models that balance autonomy and oversight. ✅ Responsible AI Development & Deployment:- Many organizations still focus on post-deployment fixes rather than AI safety by design. Alignment techniques today (RAG, constitutional AI, human feedback) remain fragile. We must shift toward "failsafe AI"—AI that degrades gracefully rather than unpredictably. ✅ AI Ethics & Governance:- AI risks—whether misinformation, deepfakes, or algorithmic bias—are no longer just theoretical. Geopolitical competition for AI dominance could further sideline ethical considerations. It is time for a convergence of policy, technical safety, and corporate governance models to ensure AI serves societal progress, not just market incentives. 👩💻 The Path Forward: A Call for Multidisciplinary Collaboration:- AI governance cannot be an afterthought. It must be woven into the DNA of AI systems—across research, regulation, and deployment. As someone deeply involved in AI governance and policy, I believe the future lies in co-regulation—where industry, academia, and policymakers collaborate proactively rather than reactively. ✨ How do we get there? 1️⃣ Bridging the gap between AI development and policy-making. 2️⃣ Building safety-aligned benchmarks for Agentic AI. 3️⃣ Embedding ethical constraints within AI architectures, not just in guidelines. 💡 AI is no longer just a tool—it is a co-pilot in decision-making, shaping economies, politics, and societies. The question is: can we govern it before it governs us? 🔎 Would love to hear your thoughts! What challenges do you see in ensuring AI remains safe, aligned, and trustworthy? #AIResearch #ResponsibleAI #AITrust #AgenticAI #Governance #AAAI2025 #AISafety #AIRegulation #EthicalAI

AI Agents Talking to Each Other Can Create Entirely New Risks Most discussions about AI safety focus on a single model interacting with a human. But what happens when AI agents start interacting with each other autonomously? A recent study called “Agents of Chaos” by researchers from Stanford University, Harvard University, and Northeastern University suggests the risks change dramatically. When AI agents collaborate, small errors can cascade into system-wide failures. Some examples from the research: 1. Minor mistakes can escalate quickly In one experiment, an agent trying to resolve a user complaint accidentally deleted an entire email server. When agents trigger other agents, the chain of actions can spiral far beyond the original task. 2. Agents can spread malicious instructions One agent shared a seemingly harmless “holiday calendar” file with another. Hidden inside were prompt-injection instructions, allowing the attacker’s control to spread across multiple agents. 3. Infinite loops can burn resources Agents can get stuck in endless back-and-forth interactions, consuming tokens, compute, and money indefinitely. 4. Accountability becomes unclear If Agent A triggers Agent B, which triggers Agent C, who is responsible when something goes wrong? Multi-agent systems create a new accountability gap. 5. Some risks may be structural The researchers argue some problems are deeper than engineering fixes. Large language models still struggle to distinguish data from commands and lack a clear sense of their own limitations. The industry is rapidly moving toward AI agents coordinating work across tools, APIs, and other agents. But most safety testing still focuses on single models operating in isolation. This research suggests the real challenge may emerge when AI systems start operating as ecosystems rather than tools. The shift from AI assistants → AI agent networks could introduce an entirely new class of operational risks. Research paper https://lnkd.in/ew7qVvVH

On current and evolving global risks of Artificial Intelligence: 1. The technical nature of AI systems poses regulatory design challenges. It is difficult to foresee all the AI permutations and combinations which makes it challenging to define the risks and safety standards or to align standards. 2. Opacity of AI systems. As not all AI modalities are well understood, it is challenging to design governance approaches. Effective guard rails must be in place to protect human rights. 3. The decentralized nature of AI applications makes difficult to track every instance and poses risks of the use by malicious actors. Open-source AI democratizes innovation but can also be put to malicious use. 4. Data, copyright, patents and cybersecurity. Cybersecurity is a dual risk of adversarial prompt injections: deliberate manipulation of the system for malicious use or the use of AI for large-scale complex cyberattacks. 5. AI divide. As investment in AI will reach $200B globally by 2025, there is a risk of a global AI divide. The biggest economic gains from AI will be in China (26% GDP boost in 2030) and North America (14.5%). 6. The proliferation of principles without accountability. In the past few years, hundreds of AI governance principles have emerged without accountability for AI-driven decision-making and adequate redress mechanisms. 7. The disproportionately large role of non-State actors and concentration of market power. As UN focuses on Member States, the enforcement depends on the governments capacity, resources and willingness to regulate. 8. Risk of inadequate inclusion. The underrepresentation of disadvantaged groups in the AI development and governance results in discriminatory or biased outputs. AI governance needs a gender and minority groups lens. 9. The dual challenges in the labour force. Large-scale AI-driven automation poses risks to the future of work. In addition, overreliance on AI systems can in the longer term result in deskilling. 10. Environmental footprint. With foundation models with trillions of parameters, the AI compute requirements are increasing the demand for hardware containing rare minerals. The need for cloud computing increases energy and water consumption needs. More info on the subject in UN white paper on AI: https://lnkd.in/e3_SbEzP

𝐀𝐈 𝐫𝐢𝐬𝐤 𝐢𝐬𝐧’𝐭 𝐨𝐧𝐞 𝐭𝐡𝐢𝐧𝐠. 𝐈𝐭’𝐬 𝟏,𝟔𝟎𝟎 𝐭𝐡𝐢𝐧𝐠𝐬. That’s not hyperbole. A new meta-review compiled over 1,600 distinct AI risks from 65 frameworks and surfaced a tough truth: most organizations are underestimating both the scope and structure of AI risk. It’s not just about bias, fairness, or hallucination. Risks emerge at different stages, from different actors, with different incentives: • Pre-deployment design decisions • Post-deployment human misuse • Model failure, misalignment, drift • Unclear accountability across teams The taxonomy distinguishes between human and AI causes, intentional and unintentional behaviors, and domain-specific vs. systemic risks. But here’s the real insight: Most AI risks don’t stem from malicious design. They emerge from fragmented ownership and unmanaged complexity. No single team sees the whole picture. Governance lives in compliance. Development lives in product. Monitoring lives in infra. And no one owns the handoffs. → Strategic takeaway: You don’t need another checklist. You need a cross-functional risk architecture. One that maps responsibility, observability, and escalation paths, before the headlines do it for you. AI systems won’t fail in one place. They’ll fail at the intersections. 𝐓𝐫𝐞𝐚𝐭 𝐀𝐈 𝐫𝐢𝐬𝐤 𝐚𝐬 𝐚 𝐜𝐡𝐞𝐜𝐤𝐛𝐨𝐱, 𝐚𝐧𝐝 𝐢𝐭 𝐰𝐢𝐥𝐥 𝐬𝐡𝐨𝐰 𝐮𝐩 𝐥𝐚𝐭𝐞𝐫 𝐚𝐬 𝐚 𝐡𝐞𝐚𝐝𝐥𝐢𝐧𝐞.

AI risk is no longer a distant theory, and OpenAI founder Sam Altman frames it into three clear categories that show why responsible AI must be addressed at both #technical and #policy levels. The first risk is misuse, where bad actors could leverage powerful AI to design #bioweapons, disrupt financial systems, or attack critical infrastructure, threats that evolve faster than traditional defenses. The second is loss of control, a lower-probability but high-impact scenario in which advanced systems fail to reliably follow #human #intent, making alignment research and safety #engineering essential at the technical level. The third is quiet dominance, where AI becomes so deeply embedded in decision-making that people and even governments over-rely on it, while its reasoning grows harder to understand, raising serious governance and #accountability concerns. Together, these risks show that technical #safeguards alone are not enough; strong policies, global coordination, transparency standards, and clear responsibility #frameworks are equally necessary to ensure AI remains a #tool that serves #humanity rather than one that subtly or suddenly undermines it. #AIRisk #ResponsibleAI #AIGovernance #AISafety #TechPolicy #FutureOfAI

⚠️Privacy Risks in AI Management: Lessons from Italy’s DeepSeek Ban⚠️ Italy’s recent ban on #DeepSeek over privacy concerns underscores the need for organizations to integrate stronger data protection measures into their AI Management System (#AIMS), AI Impact Assessment (#AIIA), and AI Risk Assessment (#AIRA). Ensuring compliance with #ISO42001, #ISO42005 (DIS), #ISO23894, and #ISO27701 (DIS) guidelines is now more material than ever. 1. Strengthening AI Management Systems (AIMS) with Privacy Controls 🔑Key Considerations: 🔸ISO 42001 Clause 6.1.2 (AI Risk Assessment): Organizations must integrate privacy risk evaluations into their AI management framework. 🔸ISO 42001 Clause 6.1.4 (AI System Impact Assessment): Requires assessing AI system risks, including personal data exposure and third-party data handling. 🔸ISO 27701 Clause 5.2 (Privacy Policy): Calls for explicit privacy commitments in AI policies to ensure alignment with global data protection laws. 🪛Implementation Example: Establish an AI Data Protection Policy that incorporates ISO27701 guidelines and explicitly defines how AI models handle user data. 2. Enhancing AI Impact Assessments (AIIA) to Address Privacy Risks 🔑Key Considerations: 🔸ISO 42005 Clause 4.7 (Sensitive Use & Impact Thresholds): Mandates defining thresholds for AI systems handling personal data. 🔸ISO 42005 Clause 5.8 (Potential AI System Harms & Benefits): Identifies risks of data misuse, profiling, and unauthorized access. 🔸ISO 27701 Clause A.1.2.6 (Privacy Impact Assessment): Requires documenting how AI systems process personally identifiable information (#PII). 🪛 Implementation Example: Conduct a Privacy Impact Assessment (#PIA) during AI system design to evaluate data collection, retention policies, and user consent mechanisms. 3. Integrating AI Risk Assessments (AIRA) to Mitigate Regulatory Exposure 🔑Key Considerations: 🔸ISO 23894 Clause 6.4.2 (Risk Identification): Calls for AI models to identify and mitigate privacy risks tied to automated decision-making. 🔸ISO 23894 Clause 6.4.4 (Risk Evaluation): Evaluates the consequences of noncompliance with regulations like #GDPR. 🔸ISO 27701 Clause A.1.3.7 (Access, Correction, & Erasure): Ensures AI systems respect user rights to modify or delete their data. 🪛 Implementation Example: Establish compliance audits that review AI data handling practices against evolving regulatory standards. ➡️ Final Thoughts: Governance Can’t Wait The DeepSeek ban is a clear warning that privacy safeguards in AIMS, AIIA, and AIRA aren’t optional. They’re essential for regulatory compliance, stakeholder trust, and business resilience. 🔑 Key actions: ◻️Adopt AI privacy and governance frameworks (ISO42001 & 27701). ◻️Conduct AI impact assessments to preempt regulatory concerns (ISO 42005). ◻️Align risk assessments with global privacy laws (ISO23894 & 27701).   Privacy-first AI shouldn't be seen just as a cost of doing business, it’s actually your new competitive advantage.

I was at Hugging Face during the critical year before and after ChatGPT's release. One thing became painfully clear: the ways AI systems can fail are exponentially more numerous than traditional software. Enterprise leaders today are under-estimating AI risks. Data privacy and hallucinations are just the tip of the iceberg. What enterprises aren't seeing: The gap between perceived and actual AI failure modes is staggering. - Enterprises think they're facing 10 potential failure scenarios…  - when the reality is closer to 100. AI risks fall into two distinct categories that require completely different approaches: Internal risks: When employees use AI tools like ChatGPT, they often inadvertently upload proprietary information. Your company's competitive edge is now potentially training competitor's models. Despite disclaimer pop-ups, this happens constantly. External risks: These are far more dangerous. When your customers interact with your AI-powered experiences, a single harmful response can destroy brand trust built over decades. Remember when Gemini's image generation missteps wiped billions off Google's market cap? Shout out to Dr. Ratinder, CTO Security and Gen AI, Pure Storage. When I got on a call with Ratinder, he very enthusiastically explained to me their super comprehensive approach: ✅ Full DevSecOps program with threat modeling, code scanning, and pen testing, secure deployment and operations ✅ Security policy generation system that enforces rules on all inputs/outputs ✅ Structured prompt engineering with 20+ techniques ✅ Formal prompt and model evaluation framework ✅ Complete logging via Splunk for traceability ✅ Third-party pen testing certification for customer trust center ✅ OWASP Top 10 framework compliance ✅ Tests for jailbreaking attempts during the development phase Their rigor is top-class… a requirement for enterprise-grade AI. For most companies, external-facing AI requires 2-3x the guardrails of internal systems. Your brand reputation simply can't afford the alternative. Ask yourself: What AI risk factors is your organization overlooking? The most dangerous ones are likely those you haven't even considered.

AI Agents Are Teaming Up… and It’s Getting Messy… and very few are talking about it! If you thought one rogue AI was risky, imagine what happens when hundreds of them start interacting. Not in isolation, but as fully autonomous agents, adapting, strategizing—and sometimes scheming. Here’s what I’m seeing, and why every AI builder should care: 1. Failure is a team sport now. We’re not just talking about bugs or hallucinations. We’re seeing miscoordination (like self-driving cars trained in different countries failing to yield), conflict (when agents optimized for different goals sabotage each other), and collusion (yes, AI systems independently learning to price-fix). 2. There are seven key risk factors under the hood: • Information asymmetries: Agents bluff, hoard info, or mislead others (intentionally or not). • Network effects: Tiny changes can ripple into massive breakdowns. • Selection pressures: Agents evolve traits like deception, aggression, or spite—depending on how they’re trained. • Destabilizing dynamics: Think flash crashes, feedback loops, and chaos—but faster. • Commitment issues: Agents can’t always promise to cooperate… or worse, they can promise and shouldn’t. • Emergent agency: Groups of agents might develop collective behavior we didn’t design or anticipate. • Security threats: Multi-agent swarms can jailbreak, phish, or break safety protocols—without needing human help. 3. What’s needed isn’t just alignment—it’s coordination. That means we need new design strategies, new oversight mechanisms, and a serious rethink of how we evaluate and deploy multi-agent systems in the real world. The future of AI isn’t just smarter models. It’s social intelligence—between agents. And that brings all the chaos, politics, and unexpected behavior you’d expect from a small nation-state… except it runs on Python. #MultiAgentAI #ArtificialIntelligence #AIBuilders #AIrisks #CooperativeAI #TechStrategy #FutureOfAI

Agentic AI Security: Risks We Can’t Ignore As agentic AI systems move from experimentation to real-world deployment, their attack surface expands rapidly. The visual highlights some of the most critical security vulnerabilities emerging in agent-based AI architectures—and why teams need to address them early. Key vulnerabilities to watch closely 🥷Token / Credential Theft ��� Secrets leaking through logs or configuration files remain one of the easiest attack vectors. 🕵️♂️Token Passthrough – Forwarding client tokens to backends without validation can cascade a single breach across systems. 🪢Rug Pull Attacks – Trusted maintainers or updates becoming malicious pose a serious supply-chain risk. 💉Prompt Injection – Hidden instructions that LLMs follow too readily; often trivial to exploit with critical impact. 🧪Tool Poisoning – Malicious commands embedded invisibly within tools or workflows. 💻Command Injection – Unfiltered inputs allowing attackers to execute arbitrary commands. ⛔️Unauthenticated Access – Optional or skipped authentication that exposes entire endpoints. The pattern is clear Most of these vulnerabilities are easy or trivial to exploit, yet their impact ranges from high to critical. Agentic AI doesn’t just generate content—it takes actions. That dramatically raises the cost of security failures. What this means for builders and leaders Treat AI agents as production-grade systems, not experiments ✔️Enforce strong authentication, token hygiene, and isolation ✔️Assume prompts, tools, and updates can be adversarial ✔️Build guardrails before increasing autonomy and scale Agentic AI is powerful, but without security-first design, it can quickly become a liability. How is your team approaching agentic AI security? #AgenticAI #AISecurity #CyberSecurity #LLM