Data Privacy Risks When Using AI Tools

This Stanford study examined how six major AI companies (Anthropic, OpenAI, Google, Meta, Microsoft, and Amazon) handle user data from chatbot conversations.  Here are the main privacy concerns. 👀 All six companies use chat data for training by default, though some allow opt-out 👀 Data retention is often indefinite, with personal information stored long-term 👀 Cross-platform data merging occurs at multi-product companies (Google, Meta, Microsoft, Amazon) 👀 Children's data is handled inconsistently, with most companies not adequately protecting minors 👀 Limited transparency in privacy policies, which are complex and hard to understand and often lack crucial details about actual practices Practical Takeaways for Acceptable Use Policy and Training for nonprofits in using generative AI: ✅ Assume anything you share will be used for training - sensitive information, uploaded files, health details, biometric data, etc. ✅ Opt out when possible - proactively disable data collection for training (Meta is the one where you cannot) ✅ Information cascades through ecosystems - your inputs can lead to inferences that affect ads, recommendations, and potentially insurance or other third parties ✅ Special concern for children's data - age verification and consent protections are inconsistent Some questions to consider in acceptable use policies and to incorporate in any training. ❓ What types of sensitive information might your nonprofit staff  share with generative AI?  ❓ Does your nonprofit currently specifically identify what is considered “sensitive information” (beyond PID) and should not be shared with GenerativeAI ? Is this incorporated into training? ❓ Are you working with children, people with health conditions, or others whose data could be particularly harmful if leaked or misused? ❓ What would be the consequences if sensitive information or strategic organizational data ended up being used to train AI models? How might this affect trust, compliance, or your mission? How is this communicated in training and policy? Across the board, the Stanford research points that developers’ privacy policies lack essential information about their practices. They recommend policymakers and developers address data privacy challenges posed by LLM-powered chatbots through comprehensive federal privacy regulation, affirmative opt-in for model training, and filtering personal information from chat inputs by default. “We need to promote innovation in privacy-preserving AI, so that user privacy isn’t an afterthought." How are you advocating for privacy-preserving AI? How are you educating your staff to navigate this challenge? https://lnkd.in/g3RmbEwD

🚨 AI Privacy Risks & Mitigations Large Language Models (LLMs), by Isabel Barberá, is the 107-page report about AI & Privacy you were waiting for! [Bookmark & share below]. Topics covered: - Background "This section introduces Large Language Models, how they work, and their common applications. It also discusses performance evaluation measures, helping readers understand the foundational aspects of LLM systems." - Data Flow and Associated Privacy Risks in LLM Systems "Here, we explore how privacy risks emerge across different LLM service models, emphasizing the importance of understanding data flows throughout the AI lifecycle. This section also identifies risks and mitigations and examines roles and responsibilities under the AI Act and the GDPR." - Data Protection and Privacy Risk Assessment: Risk Identification "This section outlines criteria for identifying risks and provides examples of privacy risks specific to LLM systems. Developers and users can use this section as a starting point for identifying risks in their own systems." - Data Protection and Privacy Risk Assessment: Risk Estimation & Evaluation "Guidance on how to analyse, classify and assess privacy risks is provided here, with criteria for evaluating both the probability and severity of risks. This section explains how to derive a final risk evaluation to prioritize mitigation efforts effectively." - Data Protection and Privacy Risk Control "This section details risk treatment strategies, offering practical mitigation measures for common privacy risks in LLM systems. It also discusses residual risk acceptance and the iterative nature of risk management in AI systems." - Residual Risk Evaluation "Evaluating residual risks after mitigation is essential to ensure risks fall within acceptable thresholds and do not require further action. This section outlines how residual risks are evaluated to determine whether additional mitigation is needed or if the model or LLM system is ready for deployment." - Review & Monitor "This section covers the importance of reviewing risk management activities and maintaining a risk register. It also highlights the importance of continuous monitoring to detect emerging risks, assess real-world impact, and refine mitigation strategies." - Examples of LLM Systems’ Risk Assessments "Three detailed use cases are provided to demonstrate the application of the risk management framework in real-world scenarios. These examples illustrate how risks can be identified, assessed, and mitigated across various contexts." - Reference to Tools, Methodologies, Benchmarks, and Guidance "The final section compiles tools, evaluation metrics, benchmarks, methodologies, and standards to support developers and users in managing risks and evaluating the performance of LLM systems." 👉 Download it below. 👉 NEVER MISS my AI governance updates: join my newsletter's 58,500+ subscribers (below). #AI #AIGovernance #Privacy #DataProtection #AIRegulation #EDPB

*Let's talk about how we use AI tools in our work and personal life without increasing the risk for accidental data leakage, breaches, or extortion* First and foremost, feeding national intelligence documents (or any sensitive docs) into an AI tool to determine which parts should remain classified is not the move (see photo below). Why? Many AI-based systems lack strong contextual decision-making, which can lead to accidental disclosure of private or classified materials from the AI tool. *When it comes to work related AI-usage we have to consider the following*: - Does your org have a policy against or for AI tool usage? What about local AI tool instances that are recommended for your team? - Does your organization use prompt protection tools to avoid accidental data leakage from your user's questions (aka prompts)? - Does the doc have secrets, proprietary data, employee data, customer data, passwords, etc embedded within it? Be careful and avoid entering this data to avoid data leakage -- redact first. - Do you secure your LLM tools with long random unique passwords + MFA? If you reuse passwords (and that password shows up in a data breach) it could lead to a hack and subsequent leak of your sensitive AI queries, work, details of an M&A that hasn't been announced etc leading to an even larger breach or data leak. *What does this mean for everyday folks who use AI for their everyday life?* You still can use AI! Just redact sensitive info before entering it into AI tools. For example: if you want to use AI tools to understand, say, a report from your doctor, I recommend removing personal details like your name, address, birthdate, etc before feeding it into AI tools to avoid accidental data leakage from the AI tool. *I predict within the next 6 months bad actors will start heavily targeting credentials for AI chat bots used by organizations to leak and extort prompt history and other sensitive questions and data* - They may attempt to leverage a reused password against your organization to gain access to your team AI tools and leak/extort your history that has sensitive data within it - They may attempt to phish individual's passwords/codes to leak or extort high net worth individuals AI chat bot history outside of the office - Bad actors may increase their targeting of AI chat bot infrastructure to encourage the tool to inadvertently leak sensitive details or proprietary info that users have entered into the tool *Actions to protect yourself and your team from AI tool risk* 1. Redact sensitive or proprietary info before entering into AI tools 2. Secure your AI tools with long random unique passwords + MFA to avoid extortion for hacked embarrassing or sensitive AI query history 3. Notice and report phishing against those AI credentials -- attackers use urgency and fear (such as "your account has been compromised, click here to secure") to get you to click and enter a password/code for that AI tool

A recent issue has emerged where private ChatGPT conversations, once shared, have become publicly searchable on Google. This is a huge red flag for HR. Conversations containing sensitive information, like employee personal details from CVs, confidential business plans, or even legal advice, are now potentially exposed. My key takeaways: ▶️ Data Privacy Nightmare: This isn't just a technical glitch; it's a massive data privacy risk. Imagine employee PII, performance review details, or internal strategy documents showing up in a public search. This could lead to serious breaches and legal repercussions under regulations like GDPR or state privacy laws. ▶️ Policy and Training Gap: The root of the problem is a lack of awareness. Employees are using AI tools without fully understanding the privacy and security implications. This is a clear indicator that your AI policy needs to be robust and your training needs to be a top priority. Do your employees know what they should and shouldn't be putting into AI tools, or sharing from them? ▶️ Mitigation is Key: 🔸Audit Your Tools: Review which AI tools your employees are using and what data they might be processing. 🔸Revise Your Policy: Update your acceptable use policy to explicitly address the use of generative AI, including what types of information are strictly forbidden from being inputted or shared. 🔸Train Your People: Conduct urgent training sessions to raise awareness about the risks of sharing conversations from AI tools. This situation highlights the critical need for a proactive approach to AI governance in HR. It's no longer just about the tech; it's about the people using it and the sensitive data they handle. What's your biggest concern about employees using generative AI?

ChatGPT is not your friend. It’s a database. In July 2025, Google indexed over 4,500 ChatGPT conversations containing sensitive personal information. Because users clicked “Share,” and the system created public URLs. Google crawled, indexed and shared them. Here’s what surfaced: 🔸 Mental illness, addiction, and abuse 🔸 Names, locations, emails, resumes 🔸 Medical histories, legal strategies All searchable, linkable and public until OpenAI intervened: ✔️ The “Discoverable” sharing feature was disabled on July 31. ✔️ They are working with Google and other search engines to remove indexed chats. ✔️ OpenAI reminded users: deleting a chat from history does not delete the public link. Millions of people, including employees and customers are confiding in AI. They believe it’s private and safe. But it isn’t. It’s recording. Indexing. Storing. And when systems designed for experimentation are used for confession, the boundaries between personal risk and enterprise liability vanish. What are the implications for Boards? 1️⃣ Regulatory risk Under GDPR: 🔹 Data subjects have the right to erase, access, and informed consent. 🔹 Shared AI conversations with personal or sensitive data may violate these rights. 🔹 AI-generated prompts could fall under automated decision-making clauses. Under the EU AI Act: 🔹 Transparency, risk classification, and human oversight are mandatory. 🔹 This incident may be classified as a high-risk system failure in healthcare, HR, legal. 2️⃣ Legal risk There is currently no legal confidentiality in AI interactions. ✔️ Anything entered into AI could be subpoenaed, discoverable in court or leaked. ✔️ Companies are liable if employees share PII, IP, or client data via chatbots. ✔️ HR, Legal, and Compliance teams must assume AI logs are discoverable records. 3️⃣ Reputational risk People assumed they were talking to a trusted tool. Instead, they ended up on Google. For enterprises using AI for: ▫️ Coaching or mental health ▫️ HR assistance ▫️ Legal or compliance advisory ▫️ Customer service … this is a trust risk. Public exposure = brand damage. 4️⃣ Operational risk Many organisations lack: 📌 AI input/output governance 📌 Policies for AI use in confidential workflows 📌 Deletion/audit protocols for AI-linked data Takeaway If employees or customers treat ChatGPT like a coach, or colleague, ensure to treat it like a legal and technical system. That means: ✅ Create AI use and data handling policies ✅ Restrict use of genAI in regulated or sensitive domains ✅ Review GDPR/AI Act exposure for all shared AI features ✅ Treat all AI interactions as auditable records ✅ Demand transparency from vendors: what is stored, shared, indexed? Until regulators catch up and new legal protections exist, assume every AI interaction is public, permanent, and admissible. #AIgovernance #Boardroom #EUAIACT #DigitalTrust #Stratedge

🚨 Using DeepSeek Poses Serious Risks to Your Privacy and Security 🚨 DeepSeek, the AI chatbot developed by a Chinese firm, has gained immense popularity recently. However, beneath its advanced capabilities lie critical security flaws, privacy risks, and potential ties to the Chinese government that make it unsafe for use. Here’s why you should think twice before using DeepSeek: 1. Major Data Breaches and Security Vulnerabilities Exposed Database: DeepSeek recently left over 1 million sensitive records, including chat logs and API keys, openly accessible due to an unsecured database. This exposed user data to potential cyberattacks and espionage. Unencrypted Data Transmission: The DeepSeek iOS app transmits sensitive user and device data without encryption, making it vulnerable to interception by malicious actors. Hardcoded Encryption Keys: Weak encryption practices, such as the use of outdated algorithms and hardcoded keys, further compromise user data security. 2. Ties to the Chinese Government Data Storage in China: DeepSeek stores user data on servers governed by Chinese law, which mandates companies to cooperate with state intelligence agencies. Hidden Code for Data Transmission: Researchers uncovered hidden programming in DeepSeek's code that can transmit user data directly to China Mobile, a state-owned telecommunications company with known ties to the Chinese government. National Security Concerns: U.S. lawmakers and cybersecurity experts have flagged DeepSeek as a tool for potential surveillance, urging bans on its use in government devices. 3. Privacy and Ethical Concerns Extensive Data Collection: DeepSeek collects detailed user information, including chat histories, device data, keystroke patterns, and even activity from other apps. This raises serious concerns about profiling and surveillance. Propaganda Risks: Investigations reveal that DeepSeek's outputs often align with Chinese government narratives, spreading misinformation and censorship on sensitive topics like Taiwan or human rights issues. 4. Dangerous Outputs and Misuse Potential Harmful Content Generation: Studies show that DeepSeek is significantly more likely than competitors to generate harmful or biased content, including extremist material and insecure code. Manipulation Risks: Its vulnerabilities make it easier for bad actors to exploit the platform for phishing scams, disinformation campaigns, and even cyberattacks. What Should You Do? Avoid using DeepSeek for any sensitive or personal information. Advocate for transparency and stricter regulations on AI tools that pose security risks. Stay informed about safer alternatives developed by companies with robust privacy protections. Your data is valuable—don’t let it fall into the wrong hands. Let’s prioritize safety and accountability in AI! 💡

In AI tools, the fine print isn’t optional. It’s everything. Recently checked out a cool new AI tool that promised awesome graphics. First red flag? No mention of data use, privacy or security on the site. Second red flag? Reading the terms of service, it said it takes no responsibility - it's all the LLMs it uses. Third red flag? Same terms say it can use the data for its own use. Fourth red flag? Same terms specifically state do not upload confidential information. Even if my content would be outward facing, I don't want to knowingly share my information to a third party who then shares it with LLMs and uses it for themselves. This was just my simple one AI tool review. Managing AI privacy risks is critical for all companies to do, no matter the size. Here are 5 tips to help manage AI risk: 1. Strengthen Your Data Governance Create a cross-functional team to develop clear policies on AI use cases. Consider third-party data access and usage, how AI will be used within the business, and if it involves sensitive data. Pro Tip: Use frameworks like NIST’s Data Privacy Framework to guide your efforts. 2. Conduct Privacy Impact Assessments (PIAs) for AI Review your existing PIA processes to determine if AI can be integrated into the assessment process. Assess AI-specific risks like bias, ethics, discrimination, and data inferences often made by AI models. 3. Train Your Team on AI Transparency Develop ongoing training programs to increase awareness of AI and how it intersects with privacy and employee roles. 4. Address Privacy Rights Challenges Posed by AI Determine how you will uphold privacy rights once data is embedded in a model. Consider how you will handle requests for access, portability, rectification, erasure, and processing restrictions. Remember, privacy notices should include provisions about how AI is used. 5. Manage Third-Party AI Vendors Carefully Ask vendors where they get their AI model, what kind of data is used to train the AI, and how often they refresh their data. Determine how vendors handle bias, inaccuracies, or underrepresentation in the AI’s outputs. Audit AI vendors and contracts regularly to identify new risks.   AI’s potential is immense, but so are the challenges it brings.   Be proactive. Build trust. Stay ahead.   Learn more in our carousel and blog link below 👇

AI systems are already creating privacy problems - and we now have the data to prove it. The AI Privacy Risks & Mitigations report from the Partnership on AI and AIID analyzes 80+ real-world incidents involving privacy failures in AI. These aren’t theoretical risks. These are documented harms happening in production systems today. It breaks down 10 core privacy risks like: → Overcollection: Found in 32% of incidents — data gathered far beyond what’s needed → Repurposing: 24% of cases involved data used for purposes users never expected → Consent Failures: 20% of systems either bypassed or poorly implemented consent → Output Leakage: 18% of incidents showed sensitive data leaking through model outputs or logs → Re-identification & Reverse Engineering: Present in 10% of incidents — often with anonymized data What stuck with me: → 60% of these failures impacted marginalized groups more heavily → Most weren’t caused by bad actors — but by poor design, short deadlines, or blind spots → Design decisions made early — like how training data is sourced — often decide the risk What I’d suggest if you’re building or reviewing AI systems: → Map how your data is collected, stored, and used and what users actually expect → Simulate a privacy red team: “Could someone misuse this model to extract PII?” → Bake in privacy assessments at every stage from dev to deployment → Review the AI Incident Database regularly as it’s a reality check for all of us!

Remember a week or so ago when I reported ChatGPT made all its shared conversations Public on Google? Guess what - Grok just did the same thing. Surprised? 370,000 private conversations between users and AI chatbots are now suddenly as public as a billboard on Times Square. 🤯 Executive Summary Elon Musk's xAI recently faced a significant privacy incident where their Grok chatbot inadvertently made hundreds of thousands of user conversations publicly searchable through Google and other search engines. The issue arose from Grok's "share" feature - when users clicked the share button to distribute conversations via email or text, the system generated unique URLs that search engines automatically indexed without any user notification. The exposed content ranged from routine requests like meal planning to deeply personal medical consultations, financial information, and even passwords. More troubling were conversations containing detailed instructions for illegal activities, including drug manufacturing and explosive creation. This breach highlights a fundamental flaw in how AI companies handle user privacy - what users believed were private interactions became permanently accessible to anyone with an internet connection. This incident represents more than just a technical oversight. It reveals the urgent need for AI companies to prioritize transparency about data handling practices. Unlike other platforms that quickly addressed similar issues, xAI has remained largely silent, raising questions about their commitment to user protection and responsible AI development. The Future Looking ahead, this breach will likely accelerate regulatory scrutiny of AI privacy practices. We can expect stricter requirements for explicit consent before making any user data publicly accessible, mandatory privacy impact assessments for AI features, and potentially significant financial penalties for companies that mishandle user information. The incident may also drive the development of privacy-by-design AI systems where user protection is built into the core architecture rather than added as an afterthought. What You Should Think About Before your next AI interaction, consider these actionable steps: Review the privacy settings of every AI tool you use, never share sensitive information like passwords or personal details with chatbots, and always read the fine print about data sharing policies. For businesses, conduct regular audits of your AI tools' privacy practices and establish clear guidelines for employee AI usage. How has this incident changed your approach to AI interactions? What privacy safeguards do you think should be mandatory for all AI platforms? 💭 Source: forbes

The world’s leading AI tools can be tricked into leaking sensitive data with just one carefully crafted prompt. As an AI advisor who works with Fortune 100 companies like PwC and Cisco, I’m seeing a whole new world of ‘AI security’ emerging. And it’s the scariest thing you’ll see this Halloween. AI's guardrails — meant to protect against hallucinations & hate — are not security guardrails! Recent research shows attackers can: 🚨 Extract personal information from AI tools 🚨 Bypass security measures with simple text prompts 🚨 Turn harmless queries into data-stealing commands 🚨 Make AI systems ignore their safety protocols The scariest part? These LLMs are already operating in everyday tools: → Google has integrated them into core search systems  → Tesla is using them to control vehicles → Microsoft has embedded them in Office tools → Robotics companies are building LLM-powered machines And it gets worse: even if you don’t have a proprietary tool, these vulnerabilities are present in AI tools you use every day. 😲 You should be concerned. All your personal (& business) data is at risk - documents you process through AI could be exposed, and automated systems could be compromised. Including things you share about your kids with family, medical or financial advisors. Every physical device with LLMs could be a gateway for hacking. These aren't hypothetical scenarios. Researchers at UC San Diego and Nanyang Technological University Singapore just demonstrated how simple prompts can trick AI into collecting and reporting personal information. The AI even disguises this breach with an invisible response – you wouldn't know your data was stolen. The industry is working on solutions, but here's the reality: we're racing to put AI everywhere before solving these fundamental security issues. ⚠️ As someone who’s been in this field for over a decade, I am a huge fan of AI's potential. But I also believe users need to understand these risks. What concerns you most about AI security in the tools you use daily? What would you not want to expose to AI? #AI #datasecurity #privacy