How to Prepare for AI Threats

🧾 Employees using AI to create fraudulent expense receipts 🤖 Fake or otherwise malicious “candidates” using Deepfake to hide their true identity on remote interviews until they get far enough in the process to hack your data 🎣 AI-powered phishing scams that are more sophisticated than ever Over the past few months, I’ve had to come to terms with the fact that this is our new reality. AI is here, and it is more powerful than ever. And HR professionals who continue to bury their head in the sand or stand by while “enabling” others without actually educating themselves are going to unleash serious risks and oversights across their company. Which means that HR professionals looking to stay on top of the increased risk introduced by AI need to lean into curiosity, education, and intentionality. For the record: I’m not anti-AI. AI has and will continue to help increase output, optimize efficiencies, and free up employees’ time to work on creative and energizing work instead of getting bogged down and burnt out by mind numbing, repetitive, and energy draining work. But it’s not without its risks. AI-powered fraud is real, and as HR professionals, it’s our jobs to educate ourselves — and our employees — on the risks involved and how to mitigate it. Not sure where to start? Consider the following: 📚 Educate yourself on the basics of what AI can do and partner with your broader HR, Legal, and #Compliance teams to create a plan to knowledge share and stay aware of new risks and AI-related cases of fraud, cyber hacking, etc (could be as simple as starting a Slack channel, signing up for a newsletter, subscribing to an AI-focused podcast — you get the point) 📑 Re-evaluate, update, and create new policies as necessary to make sure you’re addressing these new risks and policies around proper and improper AI usage at work (I’ll link our AI policy template below) 🧑💻 Re-evaluate, update, and roll out new trainings as necessary. Your hiring managers need to be aware of the increase in AI-powered candidate fraud we’re seeing across recruitment, how to spot it, and who to inform. Your employees need to know about the increased sophistication of #phishing scams and how to identify and report them For anyone looking for resources to get you started, here are a few I recommend: AI policy template: https://lnkd.in/e-F_A9hW AI training sample: https://lnkd.in/e8txAWjC AI phishing simulators: https://lnkd.in/eiux4QkN What big new scary #AI risks have you been seeing?

The Cyber Security Agency of Singapore (CSA) has published “Guidelines on Securing AI Systems,” to help system owners manage security risks in the use of AI throughout the five stages of the AI lifecycle. 1. Planning and Design: - Raise awareness and competency on security by providing training and guidance on the security risks of #AI to all personnel, including developers, system owners and senior leaders. - Conduct a #riskassessment and supplement it by continuous monitoring and a strong feedback loop. 2. Development: - Secure the #supplychain (training data, models, APIs, software libraries) - Ensure that suppliers appropriately manage risks by adhering to #security policies or internationally recognized standards. - Consider security benefits and trade-offs such as complexity, explainability, interpretability, and sensitivity of training data when selecting the appropriate model to use (#machinelearning, deep learning, #GenAI). - Identify, track and protect AI-related assets, including models, #data, prompts, logs and assessments. - Secure the #artificialintelligence development environment by applying standard infrastructure security principles like #accesscontrols and logging/monitoring, segregation of environments, and secure-by-default configurations. 3. Deployment: - Establish #incidentresponse, escalation and remediation plans. - Release #AIsystems only after subjecting them to appropriate and effective security checks and evaluation. 4. Operations and Maintenance: - Monitor and log inputs (queries, prompts and requests) and outputs to ensure they are performing as intended. - Adopt a secure-by-design approach to updates and continuous learning. - Establish a vulnerability disclosure process for users to share potential #vulnerabilities to the system. 5. End of Life: - Ensure proper data and model disposal according to relevant industry standards or #regulations.

"we present recommendations for organizations and governments engaged in establishing thresholds for intolerable AI risks. Our key recommendations include: ✔️ Design thresholds with adequate margins of safety to accommodate uncertainties in risk estimation and mitigation. ✔️Evaluate dual-use capabilities and other capability metrics, capability interactions, and model interactions through benchmarks, red team evaluations, and other best practices. ✔️Identify “minimal” and “substantial” increases in risk by comparing to appropriate base cases. ✔️Quantify the impact and likelihood of risks by identifying the types of harms and modeling the severity of their impacts. ✔️Supplement risk estimation exercises with qualitative approaches to impact assessment. ✔️Calibrate uncertainties and identify intolerable levels of risk by mapping the likelihood of intolerable outcomes to the potential levels of severity. ✔️Establish thresholds through multi-stakeholder deliberations and incentivize compliance through an affirmative safety approach. Through three case studies, we elaborate on operationalizing thresholds for some intolerable risks: ⚠️ Chemical, biological, radiological, and nuclear (CBRN) weapons, ⚠️ Evaluation Deception, and ⚠️ Misinformation. " Nada Madkour, PhD Deepika Raman, Evan R. Murphy, Krystal Jackson, Jessica Newman at the UC Berkeley Center for Long-Term Cybersecurity

AI & Practical Steps CISOs Can Take Now! Too much buzz around LLMs can paralyze security leaders. Reality is that, AI isn’t magic! So apply the same foundational security fundamentals. Here’s how to build a real AI security policy: 🔍 Discover AI Usage: Map who’s using AI, where it lives in your org, and intended use cases. 🔐 Govern Your Data: Classify & encrypt sensitive data. Know what data is used in AI tools, and where it goes. 🧠 Educate Users: Train teams on safe AI use. Teach spotting hallucinations and avoiding risky data sharing. 🛡️ Scan Models for Threats: Inspect model files for malware, backdoors, or typosquatting. Treat model files like untrusted code. 📈 Profile Risks (just like Cloud or BYOD): Create an executive-ready risk matrix. Document use cases, threats, business impact, and risk appetite. These steps aren’t flashy but they guard against real risks: data leaks, poisoning, serialization attacks, supply chain threats.

The Secure AI Lifecycle (SAIL) Framework is one of the actionable roadmaps for building trustworthy and secure AI systems. Key highlights include: • Mapping over 70 AI-specific risks across seven phases: Plan, Code, Build, Test, Deploy, Operate, Monitor • Introducing “Shift Up” security to protect AI abstraction layers like agents, prompts, and toolchains • Embedding AI threat modeling, governance alignment, and secure experimentation from day one • Addressing critical risks including prompt injection, model evasion, data poisoning, plugin misuse, and cross-domain prompt attacks • Integrating runtime guardrails, red teaming, sandboxing, and telemetry for continuous protection • Aligning with NIST AI RMF, ISO 42001, OWASP Top 10 for LLMs, and DASF v2.0 • Promoting cross-functional accountability across AppSec, MLOps, LLMOps, Legal, and GRC teams Who should take note: • Security architects deploying foundation models and AI-enhanced apps • MLOps and product teams working with agents, RAG pipelines, and autonomous workflows • CISOs aligning AI risk posture with compliance and regulatory needs • Policymakers and governance leaders setting enterprise-wide AI strategy Noteworthy aspects: • Built-in operational guidance with security embedded across the full AI lifecycle • Lifecycle-aware mitigations for risks like context evictions, prompt leaks, model theft, and abuse detection • Human-in-the-loop checkpoints, sandboxed execution, and audit trails for real-world assurance • Designed for both code and no-code AI platforms with complex dependency stacks Actionable step: Use the SAIL Framework to create a unified AI risk and security model with clear roles, security gates, and monitoring practices across teams. Consideration: Security in the AI era is more than a tech problem. It is an organizational imperative that demands shared responsibility, executive alignment, and continuous vigilance.

AI is not failing because of bad ideas; it’s "failing" at enterprise scale because of two big gaps: 👉 Workforce Preparation 👉 Data Security for AI While I speak globally on both topics in depth, today I want to educate us on what it takes to secure data for AI—because 70–82% of AI projects pause or get cancelled at POC/MVP stage (source: #Gartner, #MIT). Why? One of the biggest reasons is a lack of readiness at the data layer. So let’s make it simple - there are 7 phases to securing data for AI—and each phase has direct business risk if ignored. 🔹 Phase 1: Data Sourcing Security - Validating the origin, ownership, and licensing rights of all ingested data. Why It Matters: You can’t build scalable AI with data you don’t own or can’t trace. 🔹 Phase 2: Data Infrastructure Security - Ensuring data warehouses, lakes, and pipelines that support your AI models are hardened and access-controlled. Why It Matters: Unsecured data environments are easy targets for bad actors making you exposed to data breaches, IP theft, and model poisoning. 🔹 Phase 3: Data In-Transit Security - Protecting data as it moves across internal or external systems, especially between cloud, APIs, and vendors. Why It Matters: Intercepted training data = compromised models. Think of it as shipping cash across town in an armored truck—or on a bicycle—your choice. 🔹 Phase 4: API Security for Foundational Models - Safeguarding the APIs you use to connect with LLMs and third-party GenAI platforms (OpenAI, Anthropic, etc.). Why It Matters: Unmonitored API calls can leak sensitive data into public models or expose internal IP. This isn’t just tech debt. It’s reputational and regulatory risk. 🔹 Phase 5: Foundational Model Protection - Defending your proprietary models and fine-tunes from external inference, theft, or malicious querying. Why It Matters: Prompt injection attacks are real. And your enterprise-trained model? It’s a business asset. You lock your office at night—do the same with your models. 🔹 Phase 6: Incident Response for AI Data Breaches - Having predefined protocols for breaches, hallucinations, or AI-generated harm—who’s notified, who investigates, how damage is mitigated. Why It Matters: AI-related incidents are happening. Legal needs response plans. Cyber needs escalation tiers. 🔹 Phase 7: CI/CD for Models (with Security Hooks) - Continuous integration and delivery pipelines for models, embedded with testing, governance, and version-control protocols. Why It Matter: Shipping models like software means risk comes faster—and so must detection. Governance must be baked into every deployment sprint. Want your AI strategy to succeed past MVP? Focus and lock down the data. #AI #DataSecurity #AILeadership #Cybersecurity #FutureOfWork #ResponsibleAI #SolRashidi #Data #Leadership

💡Anyone in AI or Data building solutions? You need to read this. 🚨 Advancing AGI Safety: Bridging Technical Solutions and Governance Google DeepMind’s latest paper, "An Approach to Technical AGI Safety and Security," offers valuable insights into mitigating risks from Artificial General Intelligence (AGI). While its focus is on technical solutions, the paper also highlights the critical need for governance frameworks to complement these efforts. The paper explores two major risk categories—misuse (deliberate harm) and misalignment (unintended behaviors)—and proposes technical mitigations such as:   - Amplified oversight to improve human understanding of AI actions   - Robust training methodologies to align AI systems with intended goals   - System-level safeguards like monitoring and access controls, borrowing principles from computer security  However, technical solutions alone cannot address all risks. The authors emphasize that governance—through policies, standards, and regulatory frameworks—is essential for comprehensive risk reduction. This is where emerging regulations like the EU AI Act come into play, offering a structured approach to ensure AI systems are developed and deployed responsibly.  Connecting Technical Research to Governance:   1. Risk Categorization: The paper’s focus on misuse and misalignment aligns with regulatory frameworks that classify AI systems based on their risk levels. This shared language between researchers and policymakers can help harmonize technical and legal approaches to safety.   2. Technical Safeguards: The proposed mitigations (e.g., access controls, monitoring) provide actionable insights for implementing regulatory requirements for high-risk AI systems.   3. Safety Cases: The concept of “safety cases” for demonstrating reliability mirrors the need for developers to provide evidence of compliance under regulatory scrutiny.   4. Collaborative Standards: Both technical research and governance rely on broad consensus-building—whether in defining safety practices or establishing legal standards—to ensure AGI development benefits society while minimizing risks. Why This Matters:   As AGI capabilities advance, integrating technical solutions with governance frameworks is not just a necessity—it’s an opportunity to shape the future of AI responsibly. I'll put links to the paper below. Was this helpful for you? Let me know in the comments. Would this help a colleague? Share it. Want to discuss this with me? Yes! DM me. #AGISafety #AIAlignment #AIRegulations #ResponsibleAI #GoogleDeepMind #TechPolicy #AIEthics #3StandardDeviations

Yesterday, OpenAI shared updates on their efforts to enhance AI safety through red teaming - a structured methodology for testing AI systems to uncover risks and vulnerabilities by combining human expertise with automated approaches. See their blog post: https://lnkd.in/gMvPm5Ew (incl. pic below) OpenAI has been employing red teaming for years, and after initially relying on manual testing by external experts, their approach has evolved to include manual, automated, and mixed methods. Yesterday, they released two key papers: - a white paper on external red teaming practices (see: https://lnkd.in/gcsw6_DG) and - a research study introducing a new automated red teaming methodology (see: https://lnkd.in/gTtTH-QF). ---> 1) Human-Centered Red Teaming includes: - Diverse Team Composition: Red teams are formed based on specific testing goals, incorporating diverse expertise such as natural sciences, cybersecurity, and regional politics. Threat modeling helps prioritize areas for testing, with external experts refining the focus after initial priorities are set by internal teams. - Model Access: Red teamers are provided with model versions aligned to campaign goals. Early-stage testing can identify new risks, while later versions help evaluate planned mitigations. Multiple model versions may be tested during the process. - Guidance and Tools: Clear instructions, appropriate interfaces (e.g., APIs or consumer-facing platforms), and detailed documentation guidelines enable effective testing. These facilitate rapid evaluations, feedback collection, and simulations of real-world interactions. - Data Synthesis: Post-campaign analysis identifies whether examples align with existing policies or necessitate new safeguards. Insights from these assessments inform future automated evaluations and model updates. 2.) Automated Red Teaming: OpenAI has introduced an approach using reinforcement learning to generate diverse and effective testing scenarios. This method scales risk assessment by: - Brainstorming attack strategies (e.g., eliciting unsafe advice). - Training models to identify vulnerabilities through programmatic testing. - Rewarding diversity in simulated attacks to identify gaps beyond common patterns. * * * While OpenAI's methods demonstrate best practices for foundation model providers, businesses deploying AI systems must adopt similar strategies like Bias and Fairness Testing to avoid discrimination, Policy Alignment to uphold ethical standards, and Operational Safety to address risks like unsafe recommendations or data misuse. Without robust testing, issues can arise: customer service agents may give unsafe advice, financial tools might misinterpret queries, and educational chatbots could miss harmful inputs, undermining trust and safety.

Rogue AI isn’t a sci-fi threat. It’s a real-time enterprise risk. In 2024, a misconfigured AI agent at Serviceaide meant to streamline IT workflows in healthcare accidentally exposed the personal health data of 483,000+ patients at Catholic Health, NY. What happened? An autonomous agent accessed an unsecured Elasticsearch database without adequate safeguards. The result: 🔻 PHI leak 🔻 Federal disclosures 🔻 Reputational damage This wasn’t a system hack. It was a goal-oriented AI doing exactly what it was asked, without understanding the boundaries. Welcome to the era of agentic AI, systems that act independently to pursue objectives over time. And when those objectives are vague, or controls are weak? They improvise. An AI told to “reduce customer wait time” might start issuing refunds or escalating permissions - because it sees those as valid shortcuts to the goal. No malice. Just misalignment. How do we prevent this? ✅ Define clear, bounded objectives ✅ Enforce least-privilege access ✅ Monitor behavior in real time ✅ Intervene early when drift is detected Agentic AI is already here. The question is: Are your agents aligned, or are they already off-script? Let’s talk about making autonomous systems safer, together. Share your thoughts in the comments below. 🔁 Repost to keep this on the radar. 👤 Follow me (Anand Singh, PhD) for more insights on AI risk, data security & resilient tech strategy.