Data and Model Privacy Issues in Tech

AI reaches a milestone: privacy by design at scale Google AI and DeepMind have announced VaultGemma, a 1B parameter, open-weight model trained entirely with differential privacy (DP). Why does this matter? Most large LLMs carry inherent privacy risks: they can memorise and reproduce fragments of their training data. A serious issue if it’s a patient record, bank detail, or private correspondence. VaultGemma's training method - DP-SGD, which limits how much influence any datapoint has and adds noise to blur details - ensures no single personal data included in the training could later be exposed. The result: a mathematical guarantee of privacy, the strongest ever achieved at this scale. The opportunities In healthcare, finance, and government, the implications are immediate: 🔸 Hospitals can analyse patient data without risking disclosure. 🔸 Banks can detect fraud or assess credit risk within GDPR rules. 🔸 Governments can train models on citizen data while meeting privacy-by-design requirements. In each case, sensitive data shifts from a liability to an asset that can drive innovation. The challenges 1️⃣ Performance: VaultGemma is less accurate than the frontier LLMs, closer to the performance of GPT-3.5. This is the cost of stronger privacy: trading short-term capability for long-term protection. 2️⃣ Jurisdiction: The model guarantees privacy, but not sovereignty. Built by an American provider, it remains subject to U.S. law. Under the CLOUD Act, American authorities can compel access even to data hosted abroad. How this compares 💠 Gemini has strong capability and multimodality, but privacy protections rest on corporate policy. 💠 ChatGPT-5 leads in performance, but is closed & under U.S. jurisdiction. 💠 Claude is positioned as “safety-first,” yet its privacy controls are policy-based, not mathematical. By contrast, VaultGemma offers provable privacy. The trade-off is weaker performance and continued U.S. jurisdiction - but it moves the conversation from “trust us” to “prove it.” Leaders have now a wider choice for adopting AI: ✔️ Privacy-first model: trade accuracy for provable privacy. Suited for highly regulated sectors and SMEs needing compliance. Lower cost, limited customisation, under U.S. law. ✔️ Frontier LLMs: cutting-edge capability at scale. Privacy rests on policy, with jurisdiction split - U.S., Chinese, or EU law. Highest-priced via usage-based APIs, but with the broadest ecosystems and integrations. ✔️ Sovereign alternatives: slower today, but with greater control of data and law. Could adopt privacy-by-design methods like VaultGemma, though requiring heavy upfront investment. Higher initial cost, offset by customisation and long-term resilience. AI has reached a milestone: privacy by design is possible at scale. Leaders need to balance trust, compliance, performance, and control in their choices. #AI #ResponsibleAI #DataPrivacy #DigitalSovereignty #Boardroom

The Model That Knows Too Much: How AI Can Leak What It Learned What if your AI isn’t just generating answers… it’s recalling secrets? We tend to treat AI models like secure engines: data goes in, predictions come out, and whatever happens inside stays inside. But model inversion attacks challenge that assumption. With carefully crafted queries, an attacker doesn’t need to breach your database; they can interrogate your model until it begins to reveal fragments of the very data it was trained on. No firewall alerts. No ransomware screen. Just seemingly normal interactions… producing not-so-normal disclosures. As organizations rush to embed AI into healthcare, finance, legal systems, and enterprise platforms, the question is no longer just “Is the model accurate?” It’s “What does the model remember?” Because in the age of foundation models, sensitive data doesn’t have to be stolen from storage; it can be reconstructed from memory. And that turns your AI system into something far more than a tool. It becomes a potential leak. #Cybersecurity #ArtificialIntelligence #AIsecurity #ModelInversionAttacks #DataPrivacy #MachineLearning #AISafety #Infosec #DigitalRisk #FoundationModels #DataProtection #CyberRisk

On Protecting the Data Privacy of Large Language Models (LLMs): A Survey From the research paper: In this paper, we extensively investigate data privacy concerns within Large LLMs, specifically examining potential privacy threats from two folds: Privacy leakage and privacy attacks, and the pivotal technologies for privacy protection during various stages of LLM privacy inference, including federated learning, differential privacy, knowledge unlearning, and hardware-assisted privacy protection. Some key aspects from the paper: 1)Challenges: Given the intricate complexity involved in training LLMs, privacy protection research tends to dissect various phases of LLM development and deployment, including pre-training, prompt tuning, and inference 2) Future Directions: Protecting the privacy of LLMs throughout their creation process is paramount and requires a multifaceted approach. (i) Firstly, during data collection, minimizing the collection of sensitive information and obtaining informed consent from users are critical steps. Data should be anonymized or pseudonymized to mitigate re-identification risks. (ii) Secondly, in data preprocessing and model training, techniques such as federated learning, secure multiparty computation, and differential privacy can be employed to train LLMs on decentralized data sources while preserving individual privacy. (iii) Additionally, conducting privacy impact assessments and adversarial testing during model evaluation ensures potential privacy risks are identified and addressed before deployment. (iv)In the deployment phase, privacy-preserving APIs and access controls can limit access to LLMs, while transparency and accountability measures foster trust with users by providing insight into data handling practices. (v)Ongoing monitoring and maintenance, including continuous monitoring for privacy breaches and regular privacy audits, are essential to ensure compliance with privacy regulations and the effectiveness of privacy safeguards. By implementing these measures comprehensively throughout the LLM creation process, developers can mitigate privacy risks and build trust with users, thereby leveraging the capabilities of LLMs while safeguarding individual privacy. #privacy #llm #llmprivacy #mitigationstrategies #riskmanagement #artificialintelligence #ai #languagelearningmodels #security #risks

Before diving headfirst into AI, companies need to define what data privacy means to them in order to use GenAI safely. After decades of harvesting and storing data, many tech companies have created vast troves of the stuff - and not all of it is safe to use when training new GenAI models. Most companies can easily recognize obvious examples of Personally Identifying Information (PII) like Social Security numbers (SSNs) - but what about home addresses, phone numbers, or even information like how many kids a customer has? These details can be just as critical to ensure newly built GenAI products don’t compromise their users' privacy - or safety - but once this information has entered an LLM, it can be really difficult to excise it. To safely build the next generation of AI, companies need to consider some key issues: ⚠️Defining Sensitive Data: Companies need to decide what they consider sensitive beyond the obvious. Personally identifiable information (PII) covers more than just SSNs and contact information - it can include any data that paints a detailed picture of an individual and needs to be redacted to protect customers. 🔒Using Tools to Ensure Privacy: Ensuring privacy in AI requires a range of tools that can help tech companies process, redact, and safeguard sensitive information. Without these tools in place, they risk exposing critical data in their AI models. 🏗️ Building a Framework for Privacy: Redacting sensitive data isn’t just a one-time process; it needs to be a cornerstone of any company’s data management strategy as they continue to scale AI efforts. Since PII is so difficult to remove from an LLM once added, GenAI companies need to devote resources to making sure it doesn’t enter their databases in the first place. Ultimately, AI is only as safe as the data you feed into it. Companies need a clear, actionable plan to protect their customers - and the time to implement it is now.

If you are an organisation using AI or you are an AI developer, the Australian privacy regulator has just published some vital information about AI and your privacy obligations. Here is a summary of the new guides for businesses published today by the Office of the Australian Information Commissioner which articulate how Australian privacy law applies to AI and set out the regulator’s expectations. The first guide is aimed to help businesses comply with their privacy obligations when using commercially available AI products and help them to select an appropriate product. The second provides privacy guidance to developers using personal information to train generative AI models. GUIDE ONE: Guidance on privacy and the use of commercially available AI products Top five takeaways * Privacy obligations will apply to any personal information input into an AI system, as well as the output data generated by AI (where it contains personal information).  * Businesses should update their privacy policies and notifications with clear and transparent information about their use of AI * If AI systems are used to generate or infer personal information, including images, this is a collection of personal information and must comply with APP 3 (which deals with collection of personal info). * If personal information is being input into an AI system, APP 6 requires entities to only use or disclose the information for the primary purpose for which it was collected. * As a matter of best practice, the OAIC recommends that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools. GUIDE 2: Guidance on privacy and developing and training generative AI models Top five takeaways * Developers must take reasonable steps to ensure accuracy in generative AI models. * Just because data is publicly available or otherwise accessible does not mean it can legally be used to train or fine-tune generative AI models or systems.. * Developers must take particular care with sensitive information, which generally requires consent to be collected. * Where developers are seeking to use personal information that they already hold for the purpose of training an AI model, and this was not a primary purpose of collection, they need to carefully consider their privacy obligations. * Where a developer cannot clearly establish that a secondary use for an AI-related purpose was within reasonable expectations and related to a primary purpose, to avoid regulatory risk they should seek consent for that use and/or offer individuals a meaningful and informed ability to opt-out of such a use. https://lnkd.in/gX_FrtS9

Can LLMs be trained without remembering your private data? The Problem Most language models accidentally memorise private data - even if it appears only once.   A phone number buried in a blog post, a home address mentioned in a forum, a unique email ID in a dataset… once a model sees it, it can quietly store it. And no amount of fine-tuning can reliably “erase” it later. The Solution Google introduces VaultGemma, the first open-weights language model designed from day one not to memorise personal information What did they do differently? Instead of applying differential privacy during fine-tuning (which is too late), the team trained the entire 1B-parameter model from scratch using differential privacy. That means: - Every training example has a strictly limited impact - Gradients get clipped - Noise gets added - Unique examples become statistically invisible The result? Across 1 million tests, VaultGemma memorised zero training examples, while similar models did. Performance lands around GPT-2 level -  decent, considering the strong privacy guarantees. Yes, there’s still work ahead (such as handling private information that appears multiple times), but this is a huge step toward safer, more trustworthy AI systems. The future of responsible AI may not be the model that remembers everything, but the one that remembers only what truly matters.

A new paper from Feb 2024, last revised 24 Jun 2024, by a team at Secure and Fair AI (SAFR AI) Lab at Harvard demonstrates that even with minimal data and partial model access, powerful Membership inference attacks (MIAs) on Large Language Models (LLMs) can reveal if specific data points were used to train large language models, highlighting significant privacy risks. Problem: MIAs on LLMs allow adversaries with access to the model to determine if specific data points were part of the training set, indicating potential privacy leakage. This has risk and opportunities: - Copyright Detection: MIAs can help to verify if copyrighted data was used in training. - Machine Unlearning: MIAs can help to determine is specific personal information was used for training relevant for the right-to-be-forgotten. - Train/Test Contamination: Detecting if evaluation examples were part of the training set ensures the integrity and reliability of model assessments. - Training Dataset Extraction: Extracting training data from generative models highlights privacy vulnerabilities and informs the development of more secure AI systems. Background and Technical Overview: In a MIA, an adversary with access only to the model tries to ascertain whether a data point belongs to the model’s training data. Since the adversary only has access to the model, detecting training data implies information leakage through the model. Techniques based on Differential Privacy can prevent MIAs but at a significant cost to model accuracy, particularly for large models. Research Question: While strong MIAs exist for classifiers, given the unique training processes and complex data distributions of LLMs, it was speculated whether strong MIAs are even possible against them. The study introduces two novel MIAs for pretraining data: a neural network classifier based on model gradients and a variant using only logit access, leveraging model-stealing techniques. Results: The new methods outperform existing techniques. Even with access to less than 0.001% of the training data, along with the ability to compute model gradients, it's possible to create powerful MIAs. In particular, the findings indicate that fine-tuned models are far more susceptible to privacy attacks compared to pretrained models. Using robust MIAs, the research team extracted over 50% of the training set from fine-tuned LLMs, showcasing the potential extent of data leakage. Practical takeaway: We must limit adversaries' access to models fine-tuned on sensitive data. * * * Paper: “Pandora’s White-Box: Precise Training Data Detection and Extraction in Large Language Models” By Jeffrey G. Wang, Jason Wang, Marvin Li, Seth Neel Paper: https://lnkd.in/gTGGjRwX Blog post: https://lnkd.in/gRCJdM_q Red teaming library: https://lnkd.in/gQxEnWBv Code: https://lnkd.in/g8qpDiSE. Graphic: see paper

Google has published a whitepaper on privacy in AI, proposing a practical framework for integrating Privacy Enhancing Technologies (PETs) across the entire AI lifecycle — from data collection to training, personalization, and deployment. The paper reframes privacy from “regulatory obligation” to “product design.” PETs shouldn’t be bolted on at the end just to manage compliance risk; they should be part of the system architecture from the start. The approach is: map where personal data enters the model at each stage, identify the specific privacy risks in each of those stages, and then apply targeted protections in data handling, training, and production. The framework is built around a three-way decision: privacy, utility, and cost. Teams are expected to intentionally choose the combination of PETs that offers protection without breaking product value or user experience. The whitepaper also categorizes PETs by phase: 📃Data layer: PII removal, deduplication, anonymization, synthetic data with differential privacy. ⚙️Training: differential privacy during optimization, federated learning, MPC, trusted execution environments to reduce memorization and internal exposure. 🚀Deployment: input/output filtering, secure runtime environments, on-device processing, and computation over encrypted data to protect prompts and responses in production. Finally, the document introduces the idea of creating “well-lit paths”: reusable engineering and governance patterns that make privacy part of the core infrastructure instead of something manually reinvented by each team. It’s a useful read for anyone looking to understand, in practical terms, how to apply PETs when assessing and deploying AI models.

AI is rapidly becoming the nerve-center of how we build, sell, and serve—but that also makes it a bullseye. Before you can defend your models, you need to understand how attackers break them. Here are the five most common vectors I’m seeing in the wild: 1️⃣ Prompt Injection & Jailbreaks – Hidden instructions in seemingly harmless text or images can trick a chatbot into leaking data or taking unintended actions. 2️⃣ Data / Model Poisoning – Adversaries slip malicious samples into your training or fine-tuning set, planting logic bombs that detonate after deployment. 3️⃣ Supply-Chain Manipulation – LLMs sometimes “hallucinate” package names; attackers register those libraries so an unwary dev installs malware straight from npm or PyPI. 4️⃣ Model Theft & Extraction – Bulk-scraping outputs or abusing unsecured endpoints can replicate proprietary capabilities and drain your competitive moat. 5️⃣ Membership-Inference & Privacy Leakage – Researchers keep showing they can guess whether a sensitive record was in the training set, turning personal data into low-hanging fruit. Knowing the playbook is half the battle. Stay tuned—and start threat-modeling your AI today. 🔒🤖

I'm increasingly convinced that we need to treat "AI privacy" as a distinct field within privacy, separate from but closely related to "data privacy". Just as the digital age required the evolution of data protection laws, AI introduces new risks that challenge existing frameworks, forcing us to rethink how personal data is ingested and embedded into AI systems. Key issues include: 🔹 Mass-scale ingestion – AI models are often trained on huge datasets scraped from online sources, including publicly available and proprietary information, without individuals' consent. 🔹 Personal data embedding – Unlike traditional databases, AI models compress, encode, and entrench personal data within their training, blurring the lines between the data and the model. 🔹 Data exfiltration & exposure – AI models can inadvertently retain and expose sensitive personal data through overfitting, prompt injection attacks, or adversarial exploits. 🔹 Superinference – AI uncovers hidden patterns and makes powerful predictions about our preferences, behaviours, emotions, and opinions, often revealing insights that we ourselves may not even be aware of. 🔹 AI impersonation – Deepfake and generative AI technologies enable identity fraud, social engineering attacks, and unauthorized use of biometric data. 🔹 Autonomy & control – AI may be used to make or influence critical decisions in domains such as hiring, lending, and healthcare, raising fundamental concerns about autonomy and contestability. 🔹 Bias & fairness – AI can amplify biases present in training data, leading to discriminatory outcomes in areas such as employment, financial services, and law enforcement. To date, privacy discussions have focused on data - how it's collected, used, and stored. But AI challenges this paradigm. Data is no longer static. It is abstracted, transformed, and embedded into models in ways that challenge conventional privacy protections. If "AI privacy" is about more than just the data, should privacy rights extend beyond inputs and outputs to the models themselves? If a model learns from us, should we have rights over it? #AI #AIPrivacy #Dataprivacy #Dataprotection #AIrights #Digitalrights