How to Address Vulnerabilities in AI Code

Concerned about agentic AI risks cascading through your system? Consider these emerging smart practices which adapt existing AI governance best practices for agentic AI, reinforcing a "responsible by design" approach and encompassing the AI lifecycle end-to-end: ✅ Clearly define and audit the scope, robustness, goals, performance, and security of each agent's actions and decision-making authority. ✅ Develop "AI stress tests" and assess the resilience of interconnected AI systems ✅ Implement "circuit breakers" (a.k.a kill switches or fail-safes) that can isolate failing models and prevent contagion, limiting the impact of individual AI agent failures. ✅ Implement human oversight and observability across the system, not necessarily requiring a human-in-the-loop for each agent or decision (caveat: take a risk-based, use-case dependent approach here!). ✅ Test new agents in isolated / sand-box environments that mimic real-world interactions before productionizing ✅ Ensure teams responsible for different agents share knowledge about potential risks, understand who is responsible for interventions and controls, and document who is accountable for fixes. ✅ Implement real-time monitoring and anomaly detection to track KPIs, anomalies, errors, and deviations to trigger alerts.

The Secure AI Lifecycle (SAIL) Framework is one of the actionable roadmaps for building trustworthy and secure AI systems. Key highlights include: • Mapping over 70 AI-specific risks across seven phases: Plan, Code, Build, Test, Deploy, Operate, Monitor • Introducing “Shift Up” security to protect AI abstraction layers like agents, prompts, and toolchains • Embedding AI threat modeling, governance alignment, and secure experimentation from day one • Addressing critical risks including prompt injection, model evasion, data poisoning, plugin misuse, and cross-domain prompt attacks • Integrating runtime guardrails, red teaming, sandboxing, and telemetry for continuous protection • Aligning with NIST AI RMF, ISO 42001, OWASP Top 10 for LLMs, and DASF v2.0 • Promoting cross-functional accountability across AppSec, MLOps, LLMOps, Legal, and GRC teams Who should take note: • Security architects deploying foundation models and AI-enhanced apps • MLOps and product teams working with agents, RAG pipelines, and autonomous workflows • CISOs aligning AI risk posture with compliance and regulatory needs • Policymakers and governance leaders setting enterprise-wide AI strategy Noteworthy aspects: • Built-in operational guidance with security embedded across the full AI lifecycle • Lifecycle-aware mitigations for risks like context evictions, prompt leaks, model theft, and abuse detection • Human-in-the-loop checkpoints, sandboxed execution, and audit trails for real-world assurance • Designed for both code and no-code AI platforms with complex dependency stacks Actionable step: Use the SAIL Framework to create a unified AI risk and security model with clear roles, security gates, and monitoring practices across teams. Consideration: Security in the AI era is more than a tech problem. It is an organizational imperative that demands shared responsibility, executive alignment, and continuous vigilance.

The German Federal Office for Information Security (BSI) has published the updated version of its report on "Generative AI Models - Opportunities and Risks for Industry and Authorities". See the report here: https://lnkd.in/gRvHMDqA The report categorizes risks of LLMs into three buckets. It assigns numbers to the risks (R1-R28) as well as to countermeasures to mitigate the risks (M1-M18). The 3 risk categories are: • Risks in the context of proper use of LLMs (R1 – R11); • Risks due to misuse of LLMs (R12 – R18), • Risks resulting from attacks on LLMs (R19 – R28) Both risks and countermeasures can arise at different stages in the lifecycle of an LLM: 1.) the planning phase, 2.) the data phase, 3.) the development phase where model parameters such as architecture and size get determined, or a pre-trained model is selected, 4.) the operation phase, including training and validation. The graphics below aim to highlight 1.) when in the LLM lifecycle risks emerge and 2.) at which stage countermeasures can be sensibly implemented. The report also includes a cross-reference table (see p. 25) to provide an overview of which countermeasures reduce the probability of occurrence or the extent of damage of which risks. >>> Important Areas of Focus Recommended by the Report: <<< Educate users about the capabilities and risks of Large Language Models (LLMs), including potential data leaks, misuse, and security vulnerabilities.    Testing: Thorough testing of LLMs and their applications is crucial, possibly including red teaming to simulate attacks or misuse scenarios. Handling Sensitive Data: Assume that any data accessible to LLMs during training or operation could be exposed to users. Manage sensitive data carefully and consider using techniques like Retrieval-Augmented Generation (RAG) to implement rights and role systems. Establishing Transparency: Ensure that developers and operators disclose risks, countermeasures, residual risks, and limitations to users clearly, enhancing the explainability of LLM outputs. Auditing of Inputs and Outputs: Implement filters to clean inputs and outputs to prevent unwanted actions and allow user verification and modification of outputs. Managing Prompt Injections: Address vulnerabilities to prompt injections, which manipulate LLM behavior, by restricting application rights and implementing robust security practices. Managing Training Data: Carefully select, acquire, and preprocess training data, ensuring sensitive data is securely managed. Developing Practical Expertise: Build practical expertise through experimentation with LLMs, like conducting proof-of-concept projects, to realistically assess their capabilities and limitations. #LLMs #risk #controls #GenAI

The Cyber Security Agency of Singapore (CSA) has published “Guidelines on Securing AI Systems,” to help system owners manage security risks in the use of AI throughout the five stages of the AI lifecycle. 1. Planning and Design: - Raise awareness and competency on security by providing training and guidance on the security risks of #AI to all personnel, including developers, system owners and senior leaders. - Conduct a #riskassessment and supplement it by continuous monitoring and a strong feedback loop. 2. Development: - Secure the #supplychain (training data, models, APIs, software libraries) - Ensure that suppliers appropriately manage risks by adhering to #security policies or internationally recognized standards. - Consider security benefits and trade-offs such as complexity, explainability, interpretability, and sensitivity of training data when selecting the appropriate model to use (#machinelearning, deep learning, #GenAI). - Identify, track and protect AI-related assets, including models, #data, prompts, logs and assessments. - Secure the #artificialintelligence development environment by applying standard infrastructure security principles like #accesscontrols and logging/monitoring, segregation of environments, and secure-by-default configurations. 3. Deployment: - Establish #incidentresponse, escalation and remediation plans. - Release #AIsystems only after subjecting them to appropriate and effective security checks and evaluation. 4. Operations and Maintenance: - Monitor and log inputs (queries, prompts and requests) and outputs to ensure they are performing as intended. - Adopt a secure-by-design approach to updates and continuous learning. - Establish a vulnerability disclosure process for users to share potential #vulnerabilities to the system. 5. End of Life: - Ensure proper data and model disposal according to relevant industry standards or #regulations.

ISO 5338 has key AI risk management considerations useful to security and compliance leaders. It's a non-certifiable standard laying out best practices for the AI system lifecycle. And it’s related to ISO 42001 because control A6 from Annex A specifically mentions ISO 5338. Here are some key things to think about at every stage: INCEPTION -> Why do I need a non-deterministic system? -> What types of data will the system ingest? -> What types of outputs will it create? -> What is the sensitivity of this info? -> Any regulatory requirements? -> Any contractual ones? -> Is this cost-effective? DESIGN AND DEVELOPMENT -> What type of model? Linear regressor? Neural net? -> Does it need to talk to other systems (an agent)? -> What are the consequences of bad outputs? -> What is the source of the training data? -> How / where will data be retained? -> Will there be continuous training? -> Do we need to moderate outputs? -> Is system browsing the internet? VERIFICATION AND VALIDATION -> Confirm system meets business requirements. -> Consider external review (per NIST AI RMF). -> Do red-teaming and penetration testing. -> Do unit, integration, and UA testing DEPLOYMENT -> Would deploying  system be within our risk appetite? -> If not, who is signing off? What is the justification? -> Train users and impacted parties. -> Update shared security model. -> Publish documentation. -> Add to asset inventory. OPERATION AND MONITORING -> Do we have a vulnerability disclosure program? -> Do we have a whistleblower portal? -> How are we tracking performance? -> Model drift? CONTINUOUS VALIDATION -> Is the system still meeting our business requirements? -> If there is an incident or vulnerability, what do we do? -> What are our legal disclosure requirements? -> Should we disclose even more? -> Do regular audits. RE-EVALUATION -> Has the system exceeded our risk appetite? -> If an incident, do a root cause analysis. -> Do we need to change policies? -> Revamp procedures? RETIREMENT -> Is there business need to retain model or data? Legal? -> Delete everything we don’t need, including backups. -> Audit the deletion. Are you using ISO 5338 for AI risk management?

😅 We don’t talk about AI red teaming much today, but it’ll likely become super important as AI systems mature. Microsoft's recent white paper highlights really insightful lessons from their red teaming efforts. For those unfamiliar, AI red teaming is like ethical hacking for AI, simulating real-world attacks to uncover vulnerabilities before they can be exploited. ⛳ Key Lessons: 👉 Understand the system: Align efforts with the AI’s capabilities and application context—both simple and complex systems can pose risks. 👉 Simple attacks work: Techniques like prompt engineering and jailbreaking often reveal vulnerabilities without complex methods. 👉 Beyond benchmarks: Red teaming uncovers novel risks and context-specific vulnerabilities missed by standardized tests. 👉 Automation scales: Tools like PyRIT help automate testing, covering a broader risk landscape. 👉 Humans are crucial: Automation helps, but judgment and expertise are needed to prioritize risks and design attacks. 👉 RAI harms are nuanced: Bias and harmful content are pervasive but hard to measure, requiring careful, context-aware approaches. 👉 LLMs introduce new risks: They amplify existing vulnerabilities and bring new ones, like cross-prompt injection attacks. 👉 AI security is ongoing: It requires iterative testing, economic considerations, and strong policies for long-term safety. As AI becomes more mainstream, security will take center stage, and we’ll need stronger teams and initiatives to make it truly robust. Link: https://lnkd.in/eetMw4nG

🚨 Your AI agents are sitting ducks for attackers. Here's what nobody is talking about: while everyone's rushing to deploy AI agents in production, almost no one is securing them properly. The attack vectors are terrifying. Think about it. Your AI agent can now: Write and execute code on your servers Access your databases and APIs Process emails from unknown senders Make autonomous business decisions Handle sensitive customer data Traditional security? Useless here. Chat moderation tools were built for conversations, not for autonomous systems that can literally rewrite your infrastructure. Meta saw this coming. They built LlamaFirewall specifically for production AI agents. Not as a side project, but as the security backbone for their own agent deployments. This isn't your typical "block bad words" approach. LlamaFirewall operates at the system level with three core guardrails: PromptGuard 2 catches sophisticated injection attacks that would slip past conventional filters. State-of-the-art detection that actually works in production. Agent Alignment Checks audit the agent's reasoning process in real-time. This is revolutionary - it can detect when an agent's goals have been hijacked by malicious inputs before any damage is done. CodeShield scans every line of AI-generated code for vulnerabilities across 8 programming languages. Static analysis that happens as fast as the code is generated. Plus custom scanners you can configure for your specific threat model. The architecture is modular, so you're not locked into a one-size-fits-all solution. You can compose exactly the protection you need without sacrificing performance. The reality is stark: AI agents represent a new attack surface that most security teams aren't prepared for. Traditional perimeter security assumes humans are making the decisions. But when autonomous agents can generate code, access APIs, and process untrusted data, the threat model fundamentally changes. Organizations need to start thinking about AI agent security as a distinct discipline - not just an extension of existing security practices. This means implementing guardrails at multiple layers: input validation, reasoning auditing, output scanning, and action controls. For those looking to understand implementation details, there are technical resources emerging that cover practical approaches to AI agent security, including hands-on examples with frameworks like LlamaFirewall. The shift toward autonomous AI systems is happening whether security teams are ready or not. What's your take on AI agent security? Are you seeing these risks in your organization? For the full tutorial on Llama Firewall: Tutorial: https://lnkd.in/evUrVUb9 Huge thanks to Matan Kotick Amit Ziv for creating it! ♻️ Share to let others know it!

🛑AI Red Teaming: A Compliance Necessity for ISO 42001🛑 AI Red Teaming should no longer be considered an optional security measure, it should be a core requirement of your #AIGovernance and risk management program. Organizations implementing an #ISO42001 #AIMS must proactively assess and mitigate AI risks to comply with key requirements. AI red teaming provides a structured, adversarial testing approach to identify security vulnerabilities, model weaknesses, and compliance risks before AI failures impact your real-world operations. While red teaming can be performed manually, automation tools like Microsoft AI’s #PyRIT (Python Risk Identification Tool) make the process scalable, repeatable, and efficient.   ➡️ AI Red Teaming Aligns with ISO42001 ISO42001 mandates systematic AI risk management, emphasizing security, trustworthiness, and continuous monitoring. AI red teaming directly supports the following requirements: 1️⃣AI Risk Assessment & Treatment (Clause 6.1.2 & 6.1.3) 🔸AI red teaming identifies and evaluates risks, including adversarial vulnerabilities and model weaknesses. 🔸Supports risk treatment planning, ensuring mitigation strategies align with compliance requirements. 2️⃣AI Security & Trustworthiness (Annex A.6.2 & A.7.4) 🔸Ensures robust model validation against security threats like model evasion, prompt injection, and data poisoning. 🔸Validates data quality controls, preventing security flaws in AI training data. 3️⃣AI System Impact Assessments (Clause 6.1.4 & Annex A.5.2) 🔸Red teaming helps assess real-world risks, including bias, ethical concerns, and failure scenarios. 🔸Supports comprehensive documentation of AI risks, a key requirement under ISO 42001. 4️⃣Internal Auditing & Performance Evaluation (Clause 9.1 & 9.2) 🔸AI red teaming provides empirical testing results to support internal audits of AI system security. 🔸Ongoing adversarial stress-testing ensures AI models meet compliance benchmarks. 5️⃣Incident Response & Corrective Actions (Clause 10.2) 🔸AI red teaming generates structured post-attack reports, guiding AI incident response. 🔸Identifies corrective actions to strengthen AI governance policies and prevent future vulnerabilities. ➡️ AI Red Teaming is a Compliance Necessity Your challenge is clear: AI risk must be proactively managed, and security vulnerabilities must be continuously assessed. AI red teaming ensures organizations: 🔸Identify and mitigate AI risks before deployment. 🔸Harden AI models against real-world adversarial threats. 🔸Continuously monitor AI security and compliance gaps. 🔸Develop incident response plans for AI failures. Microsoft’s PyRIT is one option, but not the only tool. A strong AI red teaming strategy should include multiple layers of risk assessment to ensure conformance to the intent of ISO42001. A-LIGN Joseph Cortese #TheBusinessofCompliance #ComplianceAlignedtoYou