How to Manage AI User Data

How To Handle Sensitive Information in your next AI Project It's crucial to handle sensitive user information with care. Whether it's personal data, financial details, or health information, understanding how to protect and manage it is essential to maintain trust and comply with privacy regulations. Here are 5 best practices to follow: 1. Identify and Classify Sensitive Data Start by identifying the types of sensitive data your application handles, such as personally identifiable information (PII), sensitive personal information (SPI), and confidential data. Understand the specific legal requirements and privacy regulations that apply, such as GDPR or the California Consumer Privacy Act. 2. Minimize Data Exposure Only share the necessary information with AI endpoints. For PII, such as names, addresses, or social security numbers, consider redacting this information before making API calls, especially if the data could be linked to sensitive applications, like healthcare or financial services. 3. Avoid Sharing Highly Sensitive Information Never pass sensitive personal information, such as credit card numbers, passwords, or bank account details, through AI endpoints. Instead, use secure, dedicated channels for handling and processing such data to avoid unintended exposure or misuse. 4. Implement Data Anonymization When dealing with confidential information, like health conditions or legal matters, ensure that the data cannot be traced back to an individual. Anonymize the data before using it with AI services to maintain user privacy and comply with legal standards. 5. Regularly Review and Update Privacy Practices Data privacy is a dynamic field with evolving laws and best practices. To ensure continued compliance and protection of user data, regularly review your data handling processes, stay updated on relevant regulations, and adjust your practices as needed. Remember, safeguarding sensitive information is not just about compliance — it's about earning and keeping the trust of your users.

The Cybersecurity and Infrastructure Security Agency together with the National Security Agency, the Federal Bureau of Investigation (FBI), the National Cyber Security Centre, and other international organizations, published this advisory providing recommendations for organizations in how to protect the integrity, confidentiality, and availability of the data used to train and operate #artificialintelligence. The advisory focuses on three main risk areas: 1. Data #supplychain threats: Including compromised third-party data, poisoning of datasets, and lack of provenance verification. 2. Maliciously modified data: Covering adversarial #machinelearning, statistical bias, metadata manipulation, and unauthorized duplication. 3. Data drift: The gradual degradation of model performance due to changes in real-world data inputs over time. The best practices recommended include: - Tracking data provenance and applying cryptographic controls such as digital signatures and secure hashes. - Encrypting data at rest, in transit, and during processing—especially sensitive or mission-critical information. - Implementing strict access controls and classification protocols based on data sensitivity. - Applying privacy-preserving techniques such as data masking, differential #privacy, and federated learning. - Regularly auditing datasets and metadata, conducting anomaly detection, and mitigating statistical bias. - Securely deleting obsolete data and continuously assessing #datasecurity risks. This is a helpful roadmap for any organization deploying #AI, especially those working with limited internal resources or relying on third-party data.

If your team is asking “Can we use this AI tool?” You need governance.   Especially when AI systems can develop discriminatory bias, give incorrect advice, leak customer data, introduce security flaws, and perpetuate outdated assumptions about users.   AI governance programs and assessments are no longer an optional best practice.   They're on the fast track to becoming mandatory as several AI regulations roll out. Most notably for high-risk AI use. I recommend AI assessments beyond high risk use cases to also capture the privacy, security and ethical risks. Here’s how companies can conduct an AI risk assessment: ✔ Start by building an AI data inventory List every AI tool in use, including hidden ones embedded inside vendor software. Capture data inputs, decisions it makes, who has access, and outputs. ✔ Assess the decision impact Identify where wrong AI decisions could cause harm or discriminate, and review AI systems thoroughly to understand if it involves high-risk.   ✔ Examine company data sources Check whether your training data is current, representative, and free from historical bias. Confirm you have disclosures and permissions for use. ✔ Test for bias and fairness Run scenarios through AI systems with different demographic inputs and look for discrepancies in outcomes. ✔ Document everything Maintain detailed records of the assessment process, findings, and changes you make. Regulations like the EU AI Act and the Colorado AI Act have specific requirements for documenting high-risk AI usage.   ✔ Build monitoring checkpoints Set regular reviews and repeat risk assessments when new products or services are introduced or as models, vendors, business needs, or regulations change. AI oversight isn’t coming someday. It’s here.   Companies that start preparing now will be ready when the new regulations come into force. Read our full blog for more tips and to see how to put this into action 👇

The EDPB recently published a report on AI Privacy Risks and Mitigations in LLMs.   This is one of the most practical and detailed resources I've seen from the EDPB, with extensive guidance for developers and deployers. The report walks through privacy risks associated with LLMs across the AI lifecycle, from data collection and training to deployment and retirement, and offers practical tips for identifying, measuring, and mitigating risks.   Here's a quick summary of some of the key mitigations mentioned in the report:   For providers: • Fine-tune LLMs on curated, high-quality datasets and limit the scope of model outputs to relevant and up-to-date information. • Use robust anonymisation techniques and automated tools to detect and remove personal data from training data. • Apply input filters and user warnings during deployment to discourage users from entering personal data, as well as automated detection methods to flag or anonymise sensitive input data before it is processed. • Clearly inform users about how their data will be processed through privacy policies, instructions, warning or disclaimers in the user interface. • Encrypt user inputs and outputs during transmission and storage to protect data from unauthorized access. • Protect against prompt injection and jailbreaking by validating inputs, monitoring LLMs for abnormal input behaviour, and limiting the amount of text a user can input. • Apply content filtering and human review processes to flag sensitive or inappropriate outputs. • Limit data logging and provide configurable options to deployers regarding log retention. • Offer easy-to-use opt-in/opt-out options for users whose feedback data might be used for retraining.   For deployers: • Enforce strong authentication to restrict access to the input interface and protect session data. • Mitigate adversarial attacks by adding a layer for input sanitization and filtering, monitoring and logging user queries to detect unusual patterns. • Work with providers to ensure they do not retain or misuse sensitive input data. • Guide users to avoid sharing unnecessary personal data through clear instructions, training and warnings. • Educate employees and end users on proper usage, including the appropriate use of outputs and phishing techniques that could trick individuals into revealing sensitive information. • Ensure employees and end users avoid overreliance on LLMs for critical or high-stakes decisions without verification, and ensure outputs are reviewed by humans before implementation or dissemination. • Securely store outputs and restrict access to authorised personnel and systems.   This is a rare example where the EDPB strikes a good balance between practical safeguards and legal expectations. Link to the report included in the comments.   #AIprivacy #LLMs #dataprotection #AIgovernance #EDPB #privacybydesign #GDPR

13 national cyber agencies from around the world, led by #ACSC, have collaborated on a guide for secure use of a range of "AI" technologies, and it is definitely worth a read! "Engaging with Artificial Intelligence" was written with collaboration from Australian Cyber Security Centre, along with the Cybersecurity and Infrastructure Security Agency (#CISA), FBI, NSA, NCSC-UK, CCCS, NCSC-NZ, CERT NZ, BSI, INCD, NISC, NCSC-NO, CSA, and SNCC, so you would expect this to be a tome, but it's only 15 pages! It is refreshing to see that the article is not solely focused on LLMs (eg. ChatGPT), but defines Artificial Intelligence to include Machine Learning, Natural Language Processing, and Generative AI (LLMs), while acknowledging there are other sub-fields as well. The challenges identified (with actual real-world examples!) are: 🚩 Data Poisoning of an AI Model: manipulating an AI model's training data, leading to incorrect, biased, or malicious outputs 🚩 Input Manipulation Attacks: includes prompt injection and adversarial examples, where malicious inputs are used to hijack AI model outputs or cause misclassifications 🚩 Generative AI Hallucinations: generating inaccurate or factually incorrect information 🚩 Privacy and Intellectual Property Concerns: challenges in ensuring the security of sensitive data, including personal and intellectual property, within AI systems 🚩 Model Stealing Attack: creating replicas of AI models using the outputs of existing systems, raising intellectual property and privacy issues The suggested mitigations include generic (but useful!) cybersecurity advice as well as AI-specific advice: 🔐 Implement cyber security frameworks 🔐 Assess privacy and data protection impact 🔐 Enforce phishing-resistant multi-factor authentication 🔐 Manage privileged access on a need-to-know basis 🔐 Maintain backups of AI models and training data 🔐 Conduct trials for AI systems 🔐 Use secure-by-design principles and evaluate supply chains 🔐 Understand AI system limitations 🔐 Ensure qualified staff manage AI systems 🔐 Perform regular health checks and manage data drift 🔐 Implement logging and monitoring for AI systems 🔐 Develop an incident response plan for AI systems This guide is a great practical resource for users of AI systems. I would interested to know if there are any incident response plans specifically written for AI systems - are there any available from a reputable source?

We spent years locking down who can access our data. Then we gave AI agents a skeleton key. And honestly? Good. The whole point of AI agents is that they do things. They reach into your systems, pull what they need, act on your behalf. An AI agent that can't touch your data is just an expensive chatbot. But here's what's keeping me up at night as both a CIO and an attorney. A protocol called MCP — Model Context Protocol — just crossed 97 million installs. Think of it as a universal connector standard. Before MCP, getting an AI agent to talk to your CRM or your HR system required custom, expensive, one-off plumbing. MCP standardizes that connection. Any tool that speaks MCP plugs into any MCP-compatible agent. It's the USB port for AI. That's genuinely powerful. It's also why many organizations are now connected in ways nobody explicitly approved. When an AI agent has broad system access and something goes wrong — a data breach, a compliance audit, a wrongful disclosure — the first question isn't "what did the AI do?" It's "who authorized this?" If the answer is "nobody really thought about it," you have a governance failure with real legal consequences attached. This isn't inevitable. Here's where to start: 1. Inventory your connections. Know which systems have MCP-enabled access, who approved them, and what data can be accessed. This can be especially important if the AI has access to sensitive data or even basic demographic data that could cause discrimintation. 2. Audit your vendor contracts. Most were written before agentic AI existed. Agent-level access probably isn't addressed, and that gap is your exposure. Make sure you understand your liability and what controls the vendor has in place to help mitigate it.l 3. Define your authorization model. Who can approve an AI agent connecting to a sensitive system? That decision shouldn't live with a developer. 4. Build the audit trail now. If an agent accessed protected data, can you reconstruct what it touched and when? If you can't, you're not ready for the question a regulator could eventually ask. The skeleton key is the right tool. Just make sure you know which doors it's opening before someone else finds out for you. #AIGovernance #CIO #GeneralCounsel #AgenticAI #MCP #CyberSecurity #RiskManagement #LegalTech #DigitalTransformation #CISO #DataPrivacy #EnterpriseTech #AIStrategy #TechLeadership #Compliance

✳ Integrating AI, Privacy, and Information Security Governance ✳ Your approach to implementation should: 1. Define Your Strategic Context Begin by mapping out the internal and external factors impacting AI ethics, security, and privacy. Identify key regulations, stakeholder concerns, and organizational risks (ISO42001, Clause 4; ISO27001, Clause 4; ISO27701, Clause 5.2.1). Your goal should be to create unified objectives that address AI’s ethical impacts while maintaining data protection and privacy. 2. Establish a Multi-Faceted Policy Structure Policies need to reflect ethical AI use, secure data handling, and privacy safeguards. Ensure that policies clarify responsibilities for AI ethics, data security, and privacy management (ISO42001, Clause 5.2; ISO27001, Clause 5.2; ISO27701, Clause 5.3.2). Your top management must lead this effort, setting a clear tone that prioritizes both compliance and integrity across all systems (ISO42001, Clause 5.1; ISO27001, Clause 5.1; ISO27701, Clause 5.3.1). 3. Create an Integrated Risk Assessment Process Risk assessments should cover AI-specific threats (e.g., bias), security vulnerabilities (e.g., breaches), and privacy risks (e.g., PII exposure) simultaneously (ISO42001, Clause 6.1.2; ISO27001, Clause 6.1; ISO27701, Clause 5.4.1.2). By addressing these risks together, you can ensure a more comprehensive risk management plan that aligns with organizational priorities. 4. Develop Unified Controls and Documentation Documentation and controls must cover AI lifecycle management, data security, and privacy protection. Procedures must address ethical concerns and compliance requirements (ISO42001, Clause 7.5; ISO27001, Clause 7.5; ISO27701, Clause 5.5.5). Ensure that controls overlap, such as limiting access to AI systems to authorized users only, ensuring both security and ethical transparency (ISO27001, Annex A.9; ISO42001, Clause 8.1; ISO27701, Clause 5.6.3). 5. Coordinate Integrated Audits and Reviews Plan audits that evaluate compliance with AI ethics, data protection, and privacy principles together (ISO42001, Clause 9.2; ISO27001, Clause 9.2; ISO27701, Clause 5.7.2). During management reviews, analyze the performance of all integrated systems and identify improvements (ISO42001, Clause 9.3; ISO27001, Clause 9.3; ISO27701, Clause 5.7.3). 6. Leverage Technology to Support Integration Use GRC tools to manage risks across AI, information security, and privacy. Integrate AI for anomaly detection, breach prevention, and privacy safeguards (ISO42001, Clause 8.1; ISO27001, Annex A.14; ISO27701, Clause 5.6). 7. Foster an Organizational Culture of Ethics, Security, and Privacy Training programs must address ethical AI use, secure data handling, and privacy rights simultaneously (ISO42001, Clause 7.3; ISO27001, Clause 7.2; ISO27701, Clause 5.5.3). Encourage a mindset where employees actively integrate ethics, security, and privacy into their roles (ISO27701, Clause 5.5.4).

🚨 AI isn’t dangerous. Mismanaged AI is. A few weeks ago, I was in a room with a group of CEOs. We were talking about AI adoption, and I asked one question: “What’s the number one thing keeping you from using AI across your business?” Almost everyone gave the same answer. ⟶Privacy ⟶Data protection ⟶Security And they’re not wrong. Every leader has seen the headlines. ❌ Data breaches ❌ Leaks ❌ Misuse But here’s the part we don’t talk about enough: There are risks in everything we do. ⟶Hiring ⟶Launching products ⟶Entering new markets ⟶Even sending an email can carry risk The difference? We put governance and guardrails in place. AI is no different. That’s why I recommend every company assign an AI Data Officer, whether it’s a new hire or someone you train internally. Their job is simple but critical: Protect your business while you scale with AI. Here’s where to start: ✅ Control access: Who in your company is allowed to use AI ✅ Guard boundaries: What data is safe to share (Pro tip: if you wouldn’t want it public, don’t feed it in) ✅ Lock privacy: Turn off model training so your data stays yours ✅ Tighten security: MFA and strict protocols — non-negotiable ✅ Redact: Remove identifiers before uploading sensitive info The truth is: The number one fear of AI privacy is also the number one reason to build AI governance into your company now. 👉 Without discipline, AI is a risk 👉 With discipline, AI is a competitive advantage If you can’t afford to hire someone new, train someone you trust. One dedicated person managing your data is better than none. So let me ask you: Would your business feel more confident adopting AI if you had someone monitoring your data? #AI #AIstrategy #AIforleaders #AIConsultant #Leaders #Leadership

"Is our data secure when using -aaS GenAI?" A 3-level framework security leaders can use to answer: LEVEL 1: Sending just system + user prompt In the simplest case, companies like: -> OpenAI -> Microsoft -> Anthropic have an application programming interface (API) to which you can send system and user prompts. The biggest issues? -> Training: is the model improved based on what you send it? If so, this could cause data leakage. But none of the companies I mentioned train base models on your content by default. -> Retention: how long does the provider keep your prompts? Even if it isn't training on them, having your prompts sit on someone else's servers is a risk by itself. Explore zero data retention (ZDR) if your use case (healthcare, financial services) justifies it. LEVEL 2: Retrieval-augmented generation (RAG) You can use RAG to "ground" generative AI model responses in proprietary data. This improves accuracy, but you should consider: -> Are you providing context on-the-fly via LangChain? -> Or is the provider storing it (OpenAI Assistants)? In the first case, LangChain will search in whichever repository you are keeping the data, only sending needed context (and prompts) to the model provider. In the second, however, you are storing ALL context data with a vendor, so vet it appropriately. And OpenAI doesn't offer ZDR for queries to Assistants. In the case of M365 Copilot, your context data is already stored with the model provider (Microsoft in both cases), simplifying the analysis. LEVEL 3: Fine-tuning proprietary models If you need to replicate a certain style or tone of responses, fine-tuning might be the way to go. In this case you can tweak an existing proprietary model with your own data. The model provider will store both the fine-tuning data and the resulting model. So training and retention policies are key. -> Inactive deployments in Azure auto-delete in 15 days -> OpenAI retains fine-tuning data until manual deletion -> AWS Bedrock says it doesn't train base models on it You can combine fine-tuning with RAG. So the security considerations stack on top of each other. TL;DR - if you are using generative AI -as-a-Service (-aaS), know whether you are doing the below (and what the security implications are): 1. Just sending system and user prompts 2. RAG agains your own data 3. Fine-tuning models What are your biggest -aaS AI security concerns?