Understanding Risks of AI Model Usage

"With recent advancements in artificial intelligence—particularly, powerful generative models—private and public sector actors have heralded the benefits of incorporating AI more prominently into our daily lives. Frequently cited benefits include increased productivity, efficiency, and personalization. However, the harm caused by AI remains to be more fully understood. As a result of wider AI deployment and use, the number of AI harm incidents has surged in recent years, suggesting that current approaches to harm prevention may be falling short. This report argues that this is due to a limited understanding of how AI risks materialize in practice. Leveraging AI incident reports from the AI Incident Database, it analyzes how AI deployment results in harm and identifies six key mechanisms that describe this process Intentional Harm ● Harm by design ● AI misuse ● Attacks on AI systems Unintentional Harm ● AI failures ● Failures of human oversight ● Integration harm A review of AI incidents associated with these mechanisms leads to several key takeaways that should inform AI governance approaches in the future. A one-size-fits-all approach to harm prevention will fall short. This report illustrates the diverse pathways to AI harm and the wide range of actors involved. Effective mitigation requires an equally diverse response strategy that includes sociotechnical approaches. Adopting model-based approaches alone could especially neglect integration harms and failures of human oversight. To date, risk of harm correlates only weakly with model capabilities. This report illustrates many instances of harm that implicate single-purpose AI systems. Yet many policy approaches use broad model capabilities, often proxied by computing power, as a predictor for the propensity to do harm. This fails to mitigate the significant risk associated with the irresponsible design, development, and deployment of less powerful AI systems. Tracking AI incidents offers invaluable insights into real AI risks and helps build response capacity. Technical innovation, experimentation with new use cases, and novel attack strategies will result in new AI harm incidents in the future. Keeping pace with these developments requires rapid adaptation and agile responses. Comprehensive AI incident reporting allows for learning and adaptation at an accelerated pace, enabling improved mitigation strategies and identification of novel AI risks as they emerge. Incident reporting must be recognized as a critical policy tool to address AI risks." By Mia Hoffmann at Center for Security and Emerging Technology (CSET)

This is WILD! The potential for AI systems, particularly large language models (LLMs) like GPT-4, to inadvertently aid in the creation of biological threats has become a pressing concern - to the point where OpenAI has recently published fascinating research aiming to develop an early warning system that assesses the risks associated with LLM-aided biological threat creation. By comparing the capabilities of individuals with access to GPT-4 against those using only the internet, the study aimed to discern whether AI could significantly enhance the ability to access information critical for developing biological threats. The findings revealed only mild uplifts in performance metrics such as accuracy and completeness for those participants who had access to GPT-4. Although these uplifts were not statistically significant, they mark an essential first step in ongoing research and community dialogue about AI's potential risks and benefits. The study was guided by design principles that emphasise the need for human participation, comprehensive evaluation, and the comparison of AI's efficacy against existing information sources. Such a meticulous approach is critical in navigating the complexities of AI-enabled risks while minimising information hazards. From a legal standpoint, these findings intersect with the evolving regulatory framework for AI, notably the discussions surrounding the proposed AI Act in the European Union. This Act aims to categorise AI systems based on the risk they pose and establish stringent compliance requirements for high-risk AI systems. General Purpose AI (GPAI) Models such as LLMs like GPT-4 could be considered as GPAI Models with systemic risk if they are deemed capable of facilitating the creation of biological threats. This study underscores the importance of developing robust safety measures, including secure access protocols and monitoring use cases, to prevent misuse. Moreover, it highlights the need for transparency and accountability in AI development, aligning with the AI Act’s objectives to ensure that AI technologies are developed and deployed in a manner that prioritises public welfare. The evaluation's findings call for a multifaceted research agenda to better understand and contextualise the implications of AI advancements. As AI models become more sophisticated, the potential for their misuse in creating biological threats could evolve, necessitating a comprehensive body of knowledge to guide responsible development and deployment. This includes not only technical advancements but also ethical guidelines, governance frameworks, and collaborative international efforts to ensure AI serves humanity's betterment while minimising risks of misuse. The insights garnered from this study not only contribute to the scientific discourse but also offer valuable perspectives for shaping the legal landscape around AI, ensuring it advances in harmony with the principles of safety, security and ethical responsibility.

🔥 AI Security: The New Frontier of Patient Safety Cybersecurity used to mean protecting devices, networks, and data. In the age of AI, that is no longer enough. The new threat surface is the model itself. AI security now includes: • Model poisoning • Adversarial prompts • Data injection attacks • Synthetic identity creation • Algorithmic manipulation • Compromised training datasets • Unauthorized model extraction • Real-time clinical guidance distortion If your AI is compromised, your patient care is compromised. It’s that simple. Forward-looking healthcare leaders are pivoting from: “Protect the system” → to → “Protect the intelligence behind the system.” What we protect must now include: ✔️ Model integrity ✔️ Training data lineage ✔️ API security ✔️ Prompt security ✔️ Real-time monitoring of drift ✔️ Audit trails for algorithmic decisions ✔️ Red-team testing for AI vulnerabilities In 2026, AI security will become the new patient safety. Leaders who don’t understand AI risk cannot ensure clinical safety. — Khalid Turk MBA, PMP, CHCIO, FCHIME Building systems that work, teams that thrive, and cultures that endure.

Is your team still treating AI systems exactly like regular software when it comes to security? 🤔 I've been digging into NIST's draft Cyber AI Profile (IR 8596), which I think is essential reading for any GRC professional. The comment period closed last Friday, and this guidance confirms something many of us have felt for a while: AI challenges some of the core assumptions behind our traditional security frameworks. Unlike typical software which behaves predictably AI models are probabilistic and keep evolving. That means we face a new class of risks that require us to rethink our approach. A few takeaways for those of us in GRC: 💡 1️⃣ Static Checklists Don't Cut It: Because AI behavior is less predictable, relying solely on fixed checklists risks missing important threats. The guidance encourages adopting risk models designed specifically for AI's unique uncertainties. 2️⃣ New Threats Require New Defenses: Attacks like prompt injection, data poisoning, and model extraction aren't simply variations of traditional threats like malware or SQL injection. These AI-specific risks call for tailored mitigation strategies. 3️⃣ Seeing Beyond Vendor Reports: A SOC 2 report isn't enough anymore. To truly understand AI security, you have to trace data lineage, model origins, and base models. That means gaining much deeper insight into the AI supply chain. 4️⃣ Keep an Eye on AI Models Continuously: The draft stresses ongoing monitoring to catch things like model drift, unexpected behavior, and adversarial manipulation as soon as they happen. For those guiding AI risk and compliance programs, this is a strong nudge to update your frameworks. It also reinforces my conviction that the future belongs to practitioners fluent in both AI's technical landscape and sound governance principles. Although the comment period has closed, I encourage you to review the draft. Understanding this guidance now will help you prepare for the compliance landscape that's taking shape. If you're wrestling with how to handle AI's probabilistic risks, I'd be glad to swap notes on what I'm learning. 🤝 Find the draft here --> https://lnkd.in/gzxHSsQb #AIGovernance #GRC #Cybersecurity #AIrisk #NIST #RiskManagement

As organizations transition from pilots to enterprise-wide deployment of Generative and Agentic AI, it's crucial to recognize that GAI risks differ significantly from traditional software risks. Towards that, it is important to go back to basics and this publication from 2024 by National Institute of Standards and Technology (NIST)'s Generative AI Profile does a great job! 🌐 Here are the four highest-impact risks and the mitigation actions every organization should implement:- 1. Systemic Risk: Algorithmic Monocultures & Ecosystem-Level Failures When multiple industries depend on the same foundation models, a single unexpected model behavior can lead to correlated failures across the ecosystem. ⚡ Mitigation: - - Build model diversity and avoid single-model dependencies. - Maintain fallback systems and contingency workflows. - Apply stress tests that simulate sector-wide shocks. 2. Human-Originating Risks (Misuse, Over-Trust, Manipulation) Many GAI incidents stem from human behavior, including misuse, over-reliance, indirect prompt injection, and flawed assumptions. ⚡ Mitigation:- - Implement continuous user education on limitations and safe use. - Enforce access controls, privilege separation, and plugin vetting. - Maintain audit trails and logging to identify misuse early. 3. Content Integrity Risks (Hallucinations, Synthetic Media, Provenance Failure) GAI increases the scale and believability of fabricated content, from medical misinformation to deepfake-enabled harms. ⚡ Mitigation:- - Invest in content provenance, watermarking, and metadata tracking. - Require pre-deployment testing for hallucination profiles across contexts. - Use cross-model verification before high-stakes outputs are acted upon. 4. Security Risks (Prompt Injection, Data Leakage, Model Extraction) NIST highlights increasingly sophisticated attack surfaces unique to LLMs: indirect prompt injection, data extraction, and plugin-initiated compromise. ⚡ Mitigation:- - Apply secure-by-design reviews for all LLM integration points. - Red-team regularly using GAI-specific attack methods. - Log inputs/outputs via incident-ready documentation so breaches can be traced. 🔐 The bottom line:- AI risk management is not a technical afterthought, it is now a core capability. Organizations that operationalize governance, provenance, testing, and incident disclosure (NIST’s four focus pillars) will be the ones that deploy AI safely and at scale. 💬 If you’d like to explore Gen AI and Agentic AI risks, practical mitigation strategies, or how to operationalize the NIST AI RMF for your organization, feel free to comment or DM. Let’s build safer AI systems together! #AI #GenAI #AIGovernance #NIST #AIRMF #RiskManagement #AITrust #ResponsibleAI #AILeadership

International AI Safety Report 2025 A report on the state of advanced AI capabilities and risks – written by 100 AI experts including representatives nominated by 33 countries and intergovernmental organizations The International AI Safety Report is the world’s first comprehensive synthesis of current literature of the risks and capabilities of advanced AI systems. Chaired by Turing-award winning computer scientist, Yoshua Bengio, it is the culmination of work by 100 AI experts to advance a shared international understanding of the risks of advanced Artificial Intelligence (AI). The Chair is supported by an international Expert Advisory Panel made up of representatives from 30 countries, the United Nations (UN), European Union (EU), and Organization for Economic Cooperation and Development (OECD). The report does not make policy recommendations. Instead it summarises the scientific evidence on the safety of general-purpose AI to help create a shared international understanding of risks from advanced AI and how they can be mitigated. —   The report is concerned with AI risks and AI safety and focuses on identifying these risks and evaluating methods for mitigating them. It summarises the scientific evidence on 3 core questions –  - What can general-purpose AI do?  - What are risks associated with general-purpose AI?  - What mitigation techniques are there against these risks? —  Focus On - Provide scientific information that will support informed policymaking – it does not recommend specific policies - Facilitate constructive and evidence-based discussion about the uncertainty of general-purpose AI and its outcomes - Contribute to an internationally shared scientific understanding of advanced AI safety —   The report was written by a diverse group of academics, guided by world-leading experts in AI. There was no industry or government influence over the content. The secretariat organised a thorough review, which included valuable input from global civil society and industry leaders.

🚨 The Hidden Risk in GenAI PoCs: Are We Thinking Long-Term? Over the past year, I’ve seen a surge in companies experimenting with GenAI proofs of concept (PoCs). 💡 “Let’s build a chatbot!” 💡 “We should summarize our internal documents with AI!” 💡 “Can we use LLMs to generate reports automatically?” The enthusiasm is exciting—but here’s a key question we should all be asking: Where is the data coming from? A recent WSJ article highlighted some growing security risks with large language models (LLMs): ⚠️ Data exposure – Sensitive information might be unintentionally shared. ⚠️ Prompt injections – AI models could be manipulated to reveal internal data. ⚠️ Governance challenges – Lack of data tracking can create compliance risks. As organizations explore GenAI, understanding data lineage is just as important as building the models themselves. 🔹 What data is feeding the model? 🔹 Who has access to it? 🔹 Can we track where it goes? The companies that succeed in AI won’t just be the ones with the most PoCs. They’ll be the ones that build AI responsibly, with governance and security in mind. How is your organization balancing innovation with responsible AI adoption? Would love to hear your thoughts! 🚀👇

This paper is well suited for classrooms, compliance trainings and executive workshops. "An Overview of Catastrophic AI Risks" by Hendrycks, Mazeika and Woodside presents a clear framework for understanding how advanced AI could cause catastrophic or existential harm. It identifies four principal domains of concern: • Malicious use involves the intentional weaponization of AI for bioterrorism, surveillance or disinformation • AI race dynamics arise from unsafe deployment pressures in geopolitical and commercial competition • Organizational failure stems from weak safety culture, inadequate oversight or poor security practices • Rogue AIs reflect the risk of losing control over agents that deceive, seek power or deviate from intended goals Each domain is grounded in illustrative scenarios and paired with mitigation strategies, including restricted access to dual-use models, international coordination, internal and external audits, legal liability for foundation model developers and technical research into alignment and control. The authors explain their intent: “This paper is for a wide audience, unlike most of our writing, which is for empirical AI researchers. We use imagery, stories, and a simplified style to discuss the risks that advanced AIs could pose, because we think this is an important topic for everyone.” While the paper focuses on catastrophic threats, many real-world failures are more mundane. These operational risks may not be dramatic but are just as important. Below are common failure types and their corresponding mitigation strategies, drawn from professional practice: • Adversarial manipulation → Validate models, improve interpretability and detect anomalies • Bias → Use curated data, apply fairness standards and involve affected stakeholders • Over-reliance → Maintain human-in-the-loop controls and train responsible operators • Privacy risks → Enforce anonymization, ensure regulatory compliance and audit data use • Model drift → Monitor deployed models and retrain as needed • Routine misuse → Apply access controls, define usage policies and monitor threats The message is simple. Prevent the catastrophic. Govern the routine. Both require foresight, precision and accountability.

AI risk is no longer a distant theory, and OpenAI founder Sam Altman frames it into three clear categories that show why responsible AI must be addressed at both #technical and #policy levels. The first risk is misuse, where bad actors could leverage powerful AI to design #bioweapons, disrupt financial systems, or attack critical infrastructure, threats that evolve faster than traditional defenses. The second is loss of control, a lower-probability but high-impact scenario in which advanced systems fail to reliably follow #human #intent, making alignment research and safety #engineering essential at the technical level. The third is quiet dominance, where AI becomes so deeply embedded in decision-making that people and even governments over-rely on it, while its reasoning grows harder to understand, raising serious governance and #accountability concerns. Together, these risks show that technical #safeguards alone are not enough; strong policies, global coordination, transparency standards, and clear responsibility #frameworks are equally necessary to ensure AI remains a #tool that serves #humanity rather than one that subtly or suddenly undermines it. #AIRisk #ResponsibleAI #AIGovernance #AISafety #TechPolicy #FutureOfAI