Significance of AI Safety Measures

One of the most important contributions of Google DeepMind's new AGI Safety and Security paper is a clean, actionable framing of risk types. Instead of lumping all AI risks into one “doomer” narrative, they break it down into 4 clear categories- with very different implications for mitigation: 1. Misuse → The user is the adversary This isn’t the model behaving badly on its own. It’s humans intentionally instructing it to cause harm- think jailbreak prompts, bioengineering recipes, or social engineering scripts. If we don’t build strong guardrails around access, it doesn’t matter how aligned your model is. Safety = security + control 2. Misalignment → The AI is the adversary The model understands the developer’s intent- but still chooses a path that’s misaligned. It optimizes the reward signal, not the goal behind it. This is the classic “paperclip maximizer” problem, but much more subtle in practice. Alignment isn’t a static checkbox. We need continuous oversight, better interpretability, and ways to build confidence that a system is truly doing what we intend- even as it grows more capable. 3. Mistakes → The world is the adversary Sometimes the AI just… gets it wrong. Not because it’s malicious, but because it lacks the context, or generalizes poorly. This is where brittleness shows up- especially in real-world domains like healthcare, education, or policy. Don’t just test your model- stress test it. Mistakes come from gaps in our data, assumptions, and feedback loops. It's important to build with humility and audit aggressively. 4. Structural Risks → The system is the adversary These are emergent harms- misinformation ecosystems, feedback loops, market failures- that don’t come from one bad actor or one bad model, but from the way everything interacts. These are the hardest problems- and the most underfunded. We need researchers, policymakers, and industry working together to design incentive-aligned ecosystems for AI. The brilliance of this framework: It gives us language to ask better questions. Not just “is this AI safe?” But: - Safe from whom? - In what context? - Over what time horizon? We don’t need to agree on timelines for AGI to agree that risk literacy like this is step one. I’ll be sharing more breakdowns from the paper soon- this is one of the most pragmatic blueprints I’ve seen so far. 🔗Link to the paper in comments. -------- If you found this insightful, do share it with your network ♻️ Follow me (Aishwarya Srinivasan) for more AI news, insights, and educational content to keep you informed in this hyperfast AI landscape 💙

💡Anyone in AI or Data building solutions? You need to read this. 🚨 Advancing AGI Safety: Bridging Technical Solutions and Governance Google DeepMind’s latest paper, "An Approach to Technical AGI Safety and Security," offers valuable insights into mitigating risks from Artificial General Intelligence (AGI). While its focus is on technical solutions, the paper also highlights the critical need for governance frameworks to complement these efforts. The paper explores two major risk categories—misuse (deliberate harm) and misalignment (unintended behaviors)—and proposes technical mitigations such as:   - Amplified oversight to improve human understanding of AI actions   - Robust training methodologies to align AI systems with intended goals   - System-level safeguards like monitoring and access controls, borrowing principles from computer security  However, technical solutions alone cannot address all risks. The authors emphasize that governance—through policies, standards, and regulatory frameworks—is essential for comprehensive risk reduction. This is where emerging regulations like the EU AI Act come into play, offering a structured approach to ensure AI systems are developed and deployed responsibly.  Connecting Technical Research to Governance:   1. Risk Categorization: The paper’s focus on misuse and misalignment aligns with regulatory frameworks that classify AI systems based on their risk levels. This shared language between researchers and policymakers can help harmonize technical and legal approaches to safety.   2. Technical Safeguards: The proposed mitigations (e.g., access controls, monitoring) provide actionable insights for implementing regulatory requirements for high-risk AI systems.   3. Safety Cases: The concept of “safety cases” for demonstrating reliability mirrors the need for developers to provide evidence of compliance under regulatory scrutiny.   4. Collaborative Standards: Both technical research and governance rely on broad consensus-building—whether in defining safety practices or establishing legal standards—to ensure AGI development benefits society while minimizing risks. Why This Matters:   As AGI capabilities advance, integrating technical solutions with governance frameworks is not just a necessity—it’s an opportunity to shape the future of AI responsibly. I'll put links to the paper below. Was this helpful for you? Let me know in the comments. Would this help a colleague? Share it. Want to discuss this with me? Yes! DM me. #AGISafety #AIAlignment #AIRegulations #ResponsibleAI #GoogleDeepMind #TechPolicy #AIEthics #3StandardDeviations

"The most powerful AI systems are used internally for months before they are released to the public. These internal AI systems may possess capabilities significantly ahead of the public frontier, particularly in high-stakes, dual-use areas like AI research, cybersecurity, and biotechnology. This makes them a valuable asset but also a prime target for theft, misuse, and sabotage by sophisticated threat actors, including nation-states. We argue that the industry's current security measures are likely insufficient to defend against these advanced threats. Beyond external attacks, we also analyze the inherent safety risks of these systems. In the future, we expect advanced AI models deployed internally could learn harmful behaviors, leading to possible scenarios like an AI making rogue copies of itself on company servers ("internal rogue deployment"), leaking its own source code ("self-exfiltration"), or even corrupting the development of future AI models ("successor sabotage"). To address these escalating risks, this report recommends a combination of technical and policy solutions. We argue that, as the risks of AI development increase, the industry should learn from the stringent security practices common in fields like nuclear and biological research. Government, academia, and industry should combine forces to develop AI-specific security and safety measures. We also recommend that the U.S. government increase its visibility into internal AI systems through expanded evaluations and provide intelligence support to defend the industry. Proactively managing these risks is essential for fostering a robust AI industry and for safeguarding U.S. national security." By Oscar Delaney 🔸Ashwin Acharya and Institute for AI Policy and Strategy (IAPS)

A new 145 pages-paper from Google DeepMind outlines a structured approach to technical AGI safety and security, focusing on risks significant enough to cause global harm. Link to blog post & research overview, "Taking a responsible path to AGI" - Google DeepMind, 2 April 2025: https://lnkd.in/gXsV9DKP - by Anca Dragan, Rohin Shah, John "Four" Flynn and Shane Legg * * * The paper assumes for the analysis that: - AI may exceed human-level intelligence - Timelines could be short (by 2030) - AI may accelerate its own development - Progress will be continuous enough to adapt iteratively The paper argues that technical mitigations must be complemented by governance and consensus on safety standards to prevent a “race to the bottom". To tackle the challenge, the present focus needs to be on foreseeable risks in advanced foundation models (like reasoning and agentic behavior) and prioritize practical, scalable mitigations within current ML pipelines. * * * The paper outlines 4 key AGI risk areas: --> Misuse – When a human user intentionally instructs the AI to cause harm (e.g., cyberattacks). --> Misalignment – When an AI system knowingly takes harmful actions against the developer's intent (e.g., deceptive or manipulative behavior). --> Mistakes – Accidental harms caused by the AI due to lack of knowledge or situational awareness. --> Structural Risks – Systemic harms emerging from multi-agent dynamics, culture, or incentives, with no single bad actor. * * * While the paper also addresses Mistakes - accidental harms - and Structural Risks - systemic issues - recommending testing, fallback mechanisms, monitoring, regulation, transparency, and cross-sector collaboration, the focus is on Misuse and Misalignment, which present greater risk of severe harm and are more actionable through technical and procedural mitigations. * * * >> Misuse (pp. 56–70) << Goal: Prevent bad actors from accessing and exploiting dangerous AI capabilities. Mitigations: - Safety post-training and capability suppression – Section 5.3.1–5.3.3 (pp. 60–61) - Monitoring, access restrictions, and red teaming – Sections 5.4–5.5, 5.8 (pp. 62–64, 68–70) - Security controls on model weights – Section 5.6 (pp. 66–67) - Misuse safety cases and stress testing – Section 5.1, 5.8 (pp. 56, 68–70) >> Misalignment (pp. 70–108) << Goal: Ensure AI systems pursue aligned goals—not harmful ones—even if capable of misbehavior. Model-level defenses: - Amplified oversight – Section 6.1 (pp. 71–77) - Guiding model behavior via better feedback – Section 6.2 (p. 78) - Robust oversight to generalize safe behavior, including Robust training and monitoring – Sections 6.3.3–6.3.7 (pp. 82–86) - Safer Design Patterns – Section 6.5 (pp. 87–91) - Interpretability – Section 6.6 (pp. 92–101) - Alignment stress tests – Section 6.7 (pp. 102–104) - Safety cases – Section 6.8 (pp. 104–107) * * * #AGI #safety #AGIrisk #AIsecurity

AI use in 𝗔𝗡𝗬 government is 𝗡𝗢𝗧 a partisan issue - it affects 💥everyone.💥 I am just as excited about the opportunities that AI can bring as those that are leading the way. However, prioritizing AI without strong risk management opens the door WIDE to unintended consequences. There are AI Risk Management Frameworks developed (take your pick of one) that lay out clear guidelines to prevent those unintended consequences Here are a few concerns that stand out: ⚫ Speed Over Scrutiny Rushing AI into deployment can mean skipping critical evaluations. For example, NIST emphasizes iterative testing and thorough risk assessments throughout an AI system’s lifecycle. Without these, we risk rolling out systems that aren't fully understood. ⚫ Reduced Human Oversight When AI takes center stage, human judgment can get pushed to the sidelines. Most frameworks stress the importance of oversight and accountability, ensuring that AI-driven decisions remain ethical and transparent. Without clear human responsibility, who do we hold accountable when things go wrong? ⚫ Amplified Bias and Injustice AI is only as fair as the data and design behind it. We’ve already seen hiring algorithms and law enforcement tools reinforce discrimination. If bias isn’t identified and mitigated, AI could worsen existing inequities. It's not a technical issue—it’s a societal risk. ⚫ Security and Privacy Trade-offs A hasty AI rollout without strong security measures could expose critical systems to cyber threats and privacy breaches. An AI-first approach promises efficiency and innovation, but without caution, it is overflowing with risk. Yes...our government should be innovative and leverage technological breakthroughs 𝗕𝗨𝗧...and this is a 𝗕𝗜𝗚 one...it 𝗛𝗔𝗦 𝗧𝗢 𝗕𝗘 secure, transparent, and accountable. Are we prioritizing speed over safety? -------------------------------------------------------------- Opinions are my own and not the views of my employer. -------------------------------------------------------------- 👋 Chris Hockey | Manager at Alvarez & Marsal 📌 Expert in Information and AI Governance, Risk, and Compliance 🔍 Reducing compliance and data breach risks by managing data volume and relevance 🔍 Aligning AI initiatives with the evolving AI regulatory landscape ✨ Insights on: • AI Governance • Information Governance • Data Risk • Information Management • Privacy Regulations & Compliance 🔔 Follow for strategic insights on advancing information and AI governance 🤝 Connect to explore tailored solutions that drive resilience and impact

😅 We don’t talk about AI red teaming much today, but it’ll likely become super important as AI systems mature. Microsoft's recent white paper highlights really insightful lessons from their red teaming efforts. For those unfamiliar, AI red teaming is like ethical hacking for AI, simulating real-world attacks to uncover vulnerabilities before they can be exploited. ⛳ Key Lessons: 👉 Understand the system: Align efforts with the AI’s capabilities and application context—both simple and complex systems can pose risks. 👉 Simple attacks work: Techniques like prompt engineering and jailbreaking often reveal vulnerabilities without complex methods. 👉 Beyond benchmarks: Red teaming uncovers novel risks and context-specific vulnerabilities missed by standardized tests. 👉 Automation scales: Tools like PyRIT help automate testing, covering a broader risk landscape. 👉 Humans are crucial: Automation helps, but judgment and expertise are needed to prioritize risks and design attacks. 👉 RAI harms are nuanced: Bias and harmful content are pervasive but hard to measure, requiring careful, context-aware approaches. 👉 LLMs introduce new risks: They amplify existing vulnerabilities and bring new ones, like cross-prompt injection attacks. 👉 AI security is ongoing: It requires iterative testing, economic considerations, and strong policies for long-term safety. As AI becomes more mainstream, security will take center stage, and we’ll need stronger teams and initiatives to make it truly robust. Link: https://lnkd.in/eetMw4nG

Everyone is rushing to adopt #AI as quickly as possible. Few are doing much more than nodding to the potential risks, but addressing these risks will become increasingly important as AI becomes more ubiquitous, interconnected, and powerful. Researchers have created a database of 777 AI risks. You may find this excessive, but the effort is designed to provide a framework for organizations to consider and simplify their risks.  The database breaks these risks into different causal and domain categories. The causal factors include (1) Entity: Human, AI; (2) Intentionality: Intentional, Unintentional; and (3) Timing: Pre-deployment; Post-deployment. And the Domain Taxonomy of AI Risks classifies risks into seven AI risk domains: (1) Discrimination & toxicity, (2) Privacy & security, (3) Misinformation, (4) Malicious actors & misuse, (5) Human-computer interaction, (6) Socioeconomic & environmental, and (7) AI system safety, failures, & limitations. The researchers' interesting observation is that contrary to popular opinion, the risks of AI are NOT well understood or being universally addressed. One of the researchers noted, “We found that the average frameworks mentioned just 34% of the 23 risk subdomains we identified, and nearly a quarter covered less than 20%." If you'd like to learn more, the TechCrunch article does a nice job of summarizing the research: https://lnkd.in/ghpmZ4TU You can read the research report here: https://lnkd.in/gjeEwtYa And the database of AI risks is available to you here: https://airisk.mit.edu/

Mechanistic interpretability (MI) is a nascent field of study of reverse engineering neural networks. In other words, trying to find out what’s inside the proverbial “black box.” This paper by Leonard Bereska and Efstratios Gavves provides an overview of mechanistic interpretability, its techniques, and – of particular interest to me – its relevance to AI safety and alignment. The paper highlights several ways MI could potentially mitigate AI risks: -         Prevent malicious misuse by locating and erasing sensitive information stored in the model -         Reduce competitive pressures by substantiating potential threats, promoting organizational safety cultures, and supporting AI alignment through better monitoring and evaluation -         Provide safety filters for every stage of training (before, during, and after) by rigorous evaluation of artificial cognition for honesty -         Screening for deceptive behaviors As with everything within the AI space, mechanistic interpretability can’t work in isolation. “The multi-scale risk landscape calls for a balanced research portfolio to minimize risk, where research on governance, complex systems, and multi-agent simulations complements mechanistic insights and model evaluations. The perceived utility of mechanistic interpretability for AI safety largely depends on researchers’ priors regarding the likelihood of these different risk scenarios.” P.S. Shoutout to Tim Scarfe and to yet another brilliant episode of his Machine Learning Street Talk (MLST) podcast that started this whole MI chain reaction in my brain (full episode here: https://lnkd.in/eywgpKDZ)

ISO 5338 has key AI risk management considerations useful to security and compliance leaders. It's a non-certifiable standard laying out best practices for the AI system lifecycle. And it’s related to ISO 42001 because control A6 from Annex A specifically mentions ISO 5338. Here are some key things to think about at every stage: INCEPTION -> Why do I need a non-deterministic system? -> What types of data will the system ingest? -> What types of outputs will it create? -> What is the sensitivity of this info? -> Any regulatory requirements? -> Any contractual ones? -> Is this cost-effective? DESIGN AND DEVELOPMENT -> What type of model? Linear regressor? Neural net? -> Does it need to talk to other systems (an agent)? -> What are the consequences of bad outputs? -> What is the source of the training data? -> How / where will data be retained? -> Will there be continuous training? -> Do we need to moderate outputs? -> Is system browsing the internet? VERIFICATION AND VALIDATION -> Confirm system meets business requirements. -> Consider external review (per NIST AI RMF). -> Do red-teaming and penetration testing. -> Do unit, integration, and UA testing DEPLOYMENT -> Would deploying  system be within our risk appetite? -> If not, who is signing off? What is the justification? -> Train users and impacted parties. -> Update shared security model. -> Publish documentation. -> Add to asset inventory. OPERATION AND MONITORING -> Do we have a vulnerability disclosure program? -> Do we have a whistleblower portal? -> How are we tracking performance? -> Model drift? CONTINUOUS VALIDATION -> Is the system still meeting our business requirements? -> If there is an incident or vulnerability, what do we do? -> What are our legal disclosure requirements? -> Should we disclose even more? -> Do regular audits. RE-EVALUATION -> Has the system exceeded our risk appetite? -> If an incident, do a root cause analysis. -> Do we need to change policies? -> Revamp procedures? RETIREMENT -> Is there business need to retain model or data? Legal? -> Delete everything we don’t need, including backups. -> Audit the deletion. Are you using ISO 5338 for AI risk management?