Policy Framework Evaluation

Companies like Anthropic, OpenAI, and Google DeepMind have started to adopt AI safety frameworks. In our new paper, we propose a grading rubric that can be used to evaluate these frameworks. Download the paper: https://lnkd.in/e2ZnMyYT 📄 Title: A Grading Rubric for AI Safety Frameworks 🎓 Authors: Jide Alaga, Jonas Schuett, Markus Anderljung 🌎 Background In the past year, AI companies have started to adopt AI safety frameworks. This includes Anthropic’s Responsible Scaling Policy (RSP) (https://lnkd.in/eTppfSBi), OpenAI’s Preparedness Framework (https://lnkd.in/ewdWzvHW), and Google DeepMind’s Frontier Safety Framework (https://lnkd.in/dvi9eiEX). Other companies have signaled their intent to publish similar frameworks soon. At the AI Seoul Summit 2024, 16 companies including Meta, Microsoft, and xAI signed the Frontier AI Safety Commitments (https://lnkd.in/eKCJJGcp), in which they commit to publish their own frameworks by the AI Action Summit in France early 2025. 💡 What are AI safety frameworks? AI safety frameworks are risk management policies intended to keep the potential risks associated with developing and deploying frontier AI systems to an acceptable level. These frameworks typically focus on catastrophic risks (e.g. from the use of chemical or biological weapons, cyberattacks, or loss of control). They specify, among other things: (1) how developers analyze the potential ways in which AI systems could lead to catastrophic outcomes, (2) how they gather evidence about a system’s capabilities, (3) what safety measures would be adequate for a given level of capabilities, and (4) how developers intend to ensure that they adhere to the framework and maintain its effectiveness. 📋 Grading rubric To enable governments, researchers, and civil society to pass judgment on AI safety frameworks, we propose a new grading rubric. The rubric consists of seven evaluation criteria divided into three categories: (1) Effectiveness: Would the framework, if adhered to, keep risks to an acceptable level? (2) Adherence: Will the company adhere to the framework? (3) Assurance: Can third parties provide assurance that the framework would keep risks to an acceptable level and that the company will adhere to it? We also propose 21 corresponding indicators that concretize the criteria. ⭐️ Quality tiers The evaluation criteria can be graded on a scale from A (gold standard) to F (substandard). The tiers are defined in terms of (1) how much the frameworks satisfy the specified evaluation criteria, (2) how much room for improvement they leave, and (3) to what extent the demonstrated level of effort is commensurate with the stakes.

Your board just approved a $2 million security budget. New EDR. SIEM upgrade. Threat intelligence platform. Zero Trust architecture. But here's what nobody's asking: Does your policy framework actually support what you're about to build? I've watched organizations invest millions in security technology while their policy foundation—the bedrock on which everything else depends quietly crumbles beneath them. Here's the reality most security leaders don't want to acknowledge: Your policies were written for a world that no longer exists. Think about when your current policy library was created. For most organizations, it was before cloud adoption transformed their infrastructure. Before remote work became permanent. Before AI was introduced, entirely new categories of data risk were introduced. Your business has fundamentally changed. Your threat landscape has evolved beyond recognition. Your regulatory environment has expanded dramatically. Your policies? Still written for 2019. This creates a gap that's invisible until it becomes catastrophic. You're implementing a Zero Trust architecture, but your access control policies assume a castle-and-moat network. You're adopting AI tools, but your data governance policies don't address algorithmic decision-making. You're protecting a remote workforce, but your acceptable use policies were written for office workers. The technology keeps advancing. The business keeps evolving. The policies stay frozen in time. Your policies aren't just documentation that sits in a SharePoint folder. They're the constitutional foundation of your entire security program. Everything you want to enforce must first exist in policy. Every control you implement derives its authority from documented standards. Every audit, every regulatory exam, every legal proceeding will ask: What did your policies require? If that foundation is weak, outdated, or disconnected from your current reality, everything built on top of it is structurally unsound - no matter how impressive your technology stack looks. The policies don't match the reality. This is a leadership issue, not a compliance issue. The strength of your policy framework reflects the seriousness of your security commitment. Before your next security investment, ask: Does our policy framework provide the foundation for this investment to be meaningful? Can we enforce what we're about to implement? Do our documented standards reflect the security program we're trying to build? If you can't answer yes with confidence, you're building on sand. The most strategic security investment many organizations could make isn't another tool. It's about ensuring the policy foundation is strong enough to support everything else you're trying to achieve. Start there. Cyverity

"Drawing on our analysis of eight case studies prepared by independent academic and industry experts, this white paper proposes next steps to address AI evaluation and testing challenges and opportunities by: ・Synthesizing insights from the eight case studies, also published separately, and extracting lessons relevant to AI (Part 1); ・Surveying key multistakeholder initiatives that are driving AI evaluation science and practice forward (Part 2); and ・Presenting recommendations for policymakers aiming to advance the AI evaluation and testing ecosystem and strengthen AI governance (Part 3). ... While approaches to risk evaluation and testing vary significantly across the case studies, there was one consistent, top-level takeaway: evaluation frameworks always reflect trade-offs among different policy objectives, such as safety, efficiency, and innovation.   Experts across all eight fields noted that policymakers have had to weigh trade-offs in designing evaluation frameworks. These frameworks must account for both the limits of current science and the need for agility in the face of uncertainty. They likewise agreed that early design choices, often reflecting the “DNA” of the historical moment in which they’re made, as cybersecurity expert Stewart Baker described it, are important as they are difficult to scale down or undo later.  Strict, pre-deployment testing regimes—such as those used in civil aviation, medical devices, nuclear energy, and pharmaceuticals—offer strong safety assurances but can be resource-intensive and slow to adapt. These regimes often emerged in response to well-documented failures and are backed by decades of regulatory infrastructure and detailed technical standards.   In contrast, fields marked by dynamic and complex interdependencies between the tested system and its external environment—such as cybersecurity and bank stress testing—rely on more adaptive governance frameworks, where testing may be used to generate actionable insights about risk rather than primarily serve as a trigger for regulatory enforcement.   Moreover, in pharmaceuticals, where interdependencies are at play and there is emphasis on pre-deployment testing, experts highlighted a potential trade-off with post-market monitoring of downstream risks and efficacy evaluation.  These variations in approaches across domains—stemming from differences in risk profiles, types of technologies, maturity of the evaluation science, placement of expertise in the assessor ecosystem, and context in which technologies are deployed, among other factors—also inform takeaways for AI."

World Economic Forum 𝗷𝘂𝘀𝘁 𝗽𝘂𝗯𝗹𝗶𝘀𝗵𝗲𝗱 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 𝗰𝗼𝗺𝗽𝗿𝗲𝗵𝗲𝗻𝘀𝗶𝘃𝗲 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 𝗼𝗻 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝗴𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 I've seen in December. In my two recent roles, we've deployed agents that optimize digital content on marketplaces, run retail media campaigns on platforms, create replenishment POs to prevent OOS, and identify opportunities for promotions or price increases. But here's my candid observation: most of us are moving faster than our governance frameworks can handle. This report adds a new perspective to the conversation. 𝗪𝗵𝗮𝘁'𝘀 𝗶𝗻𝘀𝗶𝗱𝗲: ⬇️ 1. Technical architecture breakdown: application, orchestration, and reasoning layers—plus protocols like MCP and A2A that enable agent interoperability across enterprise systems. 2. 7-dimensional classification system: role, autonomy, authority, predictability, function, use case, and environment. This helps you understand exactly what level of risk you're dealing with. 3. Real-world evaluation framework: task success rates, completion time, tool-use accuracy, edge case robustness, and trust indicators. Finally, practical metrics for production deployment. 4. Risk assessment lifecycle: a 5-step process from defining context to managing residual risk—mapped directly to agent capabilities and deployment scenarios. 5. Progressive governance model: baseline controls for every agent (access, monitoring, testing, human oversight), with safeguards that scale as autonomy and authority increase. 6. Multi-agent ecosystems: the future isn't single agents—it's networks of agents that negotiate, transact, and collaborate. The report covers emerging risks like drift, misalignment, and cascading failures. 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗳𝗼𝗿 𝗖𝗣𝗚: ➜ Don't underestimate agents, they're not glorified chatbots; they are powerful and act on a much higher decision-making efficiency. They're making decisions on inventory, pricing, promotions, and customer data. ➜ Without classification, you can't assess risk. Without evaluation, you can't validate performance. Without governance, you're flying blind. Time to learn what's running under the hood. ➜ The framework gives you a playbook: start with low-autonomy agents, test rigorously, scale governance as capabilities grow. And don't rely on your IT and data science teams, get your hands dirty, please, even by watching and getting involved only. ➜ This isn't academic, from what I can tell, it's designed for practitioners who need to deploy safely today while preparing for multi-agent ecosystems tomorrow. The bottom line: adoption without governance is reckless. Governance without practical frameworks is paralysis. This report gives us both. Full paper is here: https://lnkd.in/eVuBJWps #AI #AIAgents #CPG #FMCG #Enterprise #Governance #Innovation

Impact evaluation is a vital component in assessing the effectiveness of development programs and policies, bridging the gap between intentions and outcomes. This “Impact Evaluation in Practice” handbook, authored by Paul J. Gertler and his team, offers a robust framework for implementing evidence-based assessments to measure the true impact of interventions. By focusing on causal relationships, it ensures that changes observed can be attributed directly to specific programs or policies, moving beyond anecdotes to provide credible, data-driven insights. The guide explores essential methodologies such as randomized controlled trials, regression discontinuity, and difference-in-differences, making complex concepts accessible to development practitioners and policymakers. It provides practical tools for integrating evaluation into program design and operations, ensuring results are actionable and policy-relevant. Real-world case studies from various global contexts illustrate how rigorous evaluations can improve resource allocation, refine program design, and scale effective interventions. This resource serves as an indispensable toolkit for those committed to accountability and learning in development. By applying its principles, practitioners and decision-makers can foster transparency, enhance program efficiency, and contribute to global knowledge on what works to reduce poverty and improve well-being.

Government interventions against ransomware are increasing. Arrests, infrastructure takedowns, sanctions, indictments, public exposure etc. But how do we assess their impact? Together with Jamie MacColl, Sophie Williams-Dunning and Bob Herczeg, we have just published a new Virtual Routes Pharos Series report: 'Assessing the Impact of Ransomware Interventions and Countermeasures: A Framework'. The research was funded by the Auswärtiges Amt (Federal Foreign Office) Germany. The starting point is simple: there is no shared, structured way to evaluate the impact of government interventions against ransomware actors. Some operations look dramatic, others are quiet; some appear decisive, others fade quickly. Without a common framework, assessments become anecdotal and episodic. So we developed one. The framework evaluates government-led interventions across four dimensions: severity, scope, longevity (and reversibility), and signalling value. It is designed for real-world use, supports graded assessment rather than false precision, and distinguishes between actor-level effects and broader ecosystem consequences. Most importantly, it makes trade-offs visible. To illustrate how it works, we apply it to cases involving REvil, Emotet, Hive, and LockBit. Grateful to many colleagues in the field for their feedback along the way. If we want cumulative learning in government counter-ransomware policy, we need a shared analytical language. This report is a step in that direction.

Effectiveness is now the real test of AML/CFT frameworks. This new Egmont Group research looks beyond technical compliance and focuses on what really matters under the FATF methodology: how Financial Intelligence Units perform in practice. The paper takes a horizontal view across Mutual Evaluation Reports and highlights recurring factors that consistently influence effectiveness ratings, particularly around: • the real use of financial intelligence by law enforcement • FIU independence, resourcing, and analytical capability • quality and timeliness of STR/SAR reporting • international cooperation under R.29 and R.40 • feedback loops between FIUs, supervisors, and reporting entities One clear takeaway: having the legal framework in place is no longer enough. Jurisdictions that perform well are those where financial intelligence is actively used to support investigations, asset tracing, and prosecutions — not where it simply exists on paper. For anyone working in FIUs, supervision, policy, or AML programme design, this is a useful reference point as the 6th round of FATF evaluations moves further into an effectiveness-driven phase. 📄 The full paper is attached to this post. #AML #CFT #FIU #FATF #EgmontGroup #FinancialIntelligence #Supervision #RegulatoryEffectiveness

An important (and short!) new policy brief from Stanford Institute for Human-Centered Artificial Intelligence (HAI) titled "Validating Claims About AI: A Policymaker's Guide". Key Takeaways (from the paper) ✅ AI companies often use benchmarks to test their systems on narrow tasks but then make sweeping claims about broad capabilities like “reasoning” or “understanding.” This gap between testing and claims is driving misguided policy decisions and investment choices. ✅ Our systematic, three-step framework helps policymakers separate legitimate AI capabilities from unsupported claims by outlining key questions to ask: What exactly is being claimed? What was actually tested? And do the two match? ✅ Even rigorous benchmarks can mislead: We demonstrate how the respected GPQA science benchmark is often used to support inflated claims about AI reasoning abilities. The issue is not just bad benchmarks; it is how results are interpreted and marketed. ✅ High-stakes decisions about AI regulation, funding, and deployment are already being made based on questionable interpretations of benchmark results. Policymakers should use this framework to demand evidence that actually supports the claims being made. The authors present here, and in the full paper (linked in the comments), a framework to help #AIGP answer important questions regarding vendor claims. Seems like an important framework for all organizations, and especially for boards and C-suites. Dominique Shelton Leipzig James (de Gaspé) Bonar, Ph.D, PCC John Barker, Esq., AIGP, CCEP, CHPC, CHRC, CHC

Not everything that counts can be counted. CSIRO’s Impact Evaluation Guide shows how to value innovation, social change, and environmental outcomes, not just the economic ones. Most evaluation frameworks stop where the spreadsheets end. They’re great at quantifying outputs, but they struggle with the intangible. Thoselong-term shifts in behaviour, policy, or ecosystems that real impact often depends on. This document fills that gap. It was designed for scientific and innovation programmes, but the lessons apply far beyond. Here’s what makes it different 👇 1️⃣ It integrates numbers and narratives The guide recognises that research and innovation rarely produce one type of value. It blends Benefit–Cost Analysis (BCA) with qualitative approaches like contribution analysis, case studies, and social network mapping, showing how to monetise what you can, and credibly describe what you can’t. 2️⃣ It offers a nine-step roadmap Instead of scattered principles, CSIRO lays out a clear nine-step process — from defining your purpose and audience to analysing benefits, testing counterfactuals, and communicating results. This structure helps you design evaluations that are comparable across projects, a huge win for funders, research institutions, and policy bodies tired of “one-off” evaluations. 3️⃣ It values what traditional frameworks overlook The guide includes detailed methods for non-market valuation such as capturing environmental and social benefits such as improved biodiversity, health, or inclusion. Few public guides go this far in explaining how to assign credible value to outcomes that don’t have a price tag. 4️⃣ It’s built for people who straddle two worlds This guide is ideal for: Research organisations and innovation agencies that need to demonstrate real-world value to funders. Government evaluators and policy analysts who want to link scientific outputs to public good outcomes. M&E professionals tired of frameworks that ignore systems complexity or long-term change. If you’ve ever been asked to “prove impact” in a context where attribution is impossible, this guide gives you the language and structure to do it with integrity. --- 🔥 Join my FREE mailing list to get content straight in your inbox Sign up here: https://lnkd.in/ec8mqV2M #ImpactEvaluation

Enhancing Public Policy Evaluation for Better Governance Excited to share the OECD Public Policy Evaluation Toolkit, a comprehensive resource designed to help policymakers, evaluators, and government officials improve policy evaluation practices. This toolkit offers practical guidance on strengthening institutional frameworks, ensuring high-quality evaluations, and enhancing the impact of evidence-based policymaking. Key highlights: ✅ Institutionalizing public policy evaluation for better accountability ✅ Improving the quality and standards of evaluations ✅ Strengthening the role of evaluations in decision-making and fiscal planning ✅ Encouraging a culture of learning and continuous improvement Public policy evaluation is crucial for transparency, efficiency, and better governance. By adopting best practices and leveraging international experiences, we can build stronger, evidence-informed policies that drive real impact. #PublicPolicy #Governance #Evaluation #EvidenceBasedDecisionMaking