ChatGPT in the Organization: Top Security Risks and Mitigations

What Is ChatGPT? 

ChatGPT is an online generative AI tool developed by OpenAI, based on the company’s GPT (Generative Pre-trained Transformer) series of large language models (LLMs). It’s designed to understand and generate human-like text, enabling it to engage in conversational exchanges, answer questions, write content, generate code, based on its analysis of vast amounts of textual data.

The use of ChatGPT and similar generative AI tools is rapidly growing. According to research by SalesForce, over 45% of the U.S. population is using generative AI tools, and 75% of users are looking to automate tasks at work and support work communications. At the same time, the capability of ChatGPT to process and generate human-like text can be exploited for malicious purposes, raising the stakes for cybersecurity. 

From crafting sophisticated phishing emails that can fool even the vigilant eye, to generating code that could be used in cyber attacks, the potential misuse of ChatGPT highlights the dual-edged nature of advanced AI technologies. As these models become more integrated into critical business operations, the potential for data breaches and privacy violations increases, necessitating stringent security measures to safeguard sensitive information, and ensure that advanced AI capabilities do not translate into security vulnerabilities.

How Are Organizations Using Large Language Models (LLMs) Today?

LLMs, such as the models powering ChatGPT, are revolutionizing the way organizations operate across industries. Here are some of the enterprise use cases for tools like ChatGPT:

• Customer support automation: Many companies are leveraging LLMs to enhance their customer service. By integrating these models into their customer support systems, businesses can provide instant, 24/7 responses to customer inquiries.
• Content creation and management: LLMs assist in generating and managing content at scale. From drafting articles and reports to creating marketing materials and product descriptions, these models help streamline content creation processes.
• Code generation and review: In the field of software development, LLMs are being used to generate code snippets, review code for errors, and suggest improvements and optimizations.
• Language translation: Organizations with a global presence use LLMs for real-time translation services, enabling seamless communication across different languages and cultures. This helps businesses to expand their reach and cater to a broader audience.
• Personalized marketing campaigns: By analyzing customer data, LLMs can help tailor marketing messages and campaigns to individual preferences and behaviors.
• Data analysis: LLMs can process and analyze large datasets to extract insights and trends. Businesses use this capability to inform decision-making, identify market opportunities, and optimize operations.
• Educational tools and resources: Educational institutions and eLearning platforms are adopting LLMs to create personalized learning experiences. These models can generate study materials, quizzes, and even provide tutoring on a wide range of subjects.

Tips from the expert

Steve Moore is Vice President and Chief Security Strategist at Exabeam, helping drive solutions for threat detection and advising customers on security programs and breach response. He is the host of the “The New CISO Podcast,” a Forbes Tech Council member, and Co-founder of TEN18 at Exabeam.

In my experience, here are advanced strategies for securely deploying and managing tools like ChatGPT in enterprise environments:

Deploy AI-generated phishing detection tools
Train internal systems to recognize AI-generated phishing emails and social engineering attempts. Pair these with employee awareness programs to reduce the risk of falling victim to sophisticated, AI-generated attacks.

Implement an LLM usage policy across the organization
Develop a clear policy governing the use of generative AI tools like ChatGPT. Include guidelines on acceptable use cases, prohibited activities, and processes for vetting third-party integrations. Ensure employees understand the risks of using LLMs to process sensitive data.

Use a private LLM instance for sensitive environments
For enterprises handling confidential or regulated data, consider deploying private, self-hosted versions of LLMs. These instances operate within your network and offer full control over data handling and security configurations, eliminating risks associated with public APIs.

Integrate LLM usage with SIEM systems
Monitor LLM interactions using SIEM platforms like Exabeam to detect unusual activity, such as frequent or suspicious queries. This helps to identify potential misuse or malicious behavior involving AI-powered tools.

Regularly update and secure plugins and APIs
Ensure that plugins and APIs connected to ChatGPT are updated to the latest versions and follow secure coding practices. Conduct vulnerability assessments on plugins and restrict the use of custom GPTs to vetted and approved configurations.

What Are the Security Risks of LLMs in Organizations?

There are several security risks and challenges that arise when using large language models like ChatGPT.

Exposure of Sensitive Data

Sensitive data exposure is a critical concern with LLMs, as they process and incorporate information from their training datasets. In scenarios where these models inadvertently recall and reproduce details from those datasets, it can lead to unintended data leakage, compromising privacy and security.

Malicious queries designed to extract confidential information pose a significant risk, especially when LLMs are integrated into public-facing services. Implementing stringent data handling and privacy policies is pivotal in mitigating such vulnerabilities.

LLM Injection Attacks

LLM injection attacks involve adversaries manipulating the input to the model in order to trigger specific, often unauthorized responses, or to cause the system to behave in unintended ways. These attacks exploit the model’s ability to generate content based on the input it receives, potentially leading to the exposure of sensitive information, execution of harmful actions, or dissemination of misleading content. 

In an organizational context, this could result in generative AI tools being exploited by employees or external threat actors, exposing the organization to risk. To combat these threats, it’s critical to implement robust input validation and sanitization processes. Monitoring and logging all interactions with LLMs can also help identify and mitigate injection attempts.

Data Poisoning

Data poisoning is a technique where attackers deliberately feed misleading or malicious data into the model’s training set, aiming to corrupt its learning process and influence its future outputs. This can lead to the model generating incorrect, biased, or harmful content. In the context of LLMs, this could manifest as the model producing outputs that are subtly skewed in favor of an attacker’s agenda, or outputs that include hidden malicious content or advice.

Preventing data poisoning requires rigorous data validation and curation processes. It’s important to carefully monitor and control the data used for training LLMs, ensuring it comes from reputable sources. Additionally, continual reevaluation and updating of the model with clean, verified data can help mitigate the effects of any previously ingested poisoned data. Organizations must ensure that LLM providers, such as OpenAI, are taking the appropriate measures to prevent data poisoning.

Insecure Plugin Design

ChatGPT provides access to thousands of plugins and custom GPTs created by individuals around the world. When used in an organization, there is a risk of integrating plugins or add-ons that have not been adequately secured. 

Insecure plugin design can lead to serious security vulnerabilities. In theory, these vulnerabilities could allow attackers to bypass security controls, execute arbitrary code, or access sensitive information.

How Can LLMs Like ChatGPT Be Used for Criminal Activity? 

Enhancement of Cybercriminal Skills

LLMs provide cyber criminals easy access to sophisticated hacking techniques and knowledge. Adversaries can query these models for information on exploiting vulnerabilities, escalating the threat level they pose. In addition, attackers seeking to penetrate organizational systems can use LLMs to quickly learn and generate code or configuration for practically any software system.

Use in Social Engineering Attacks

LLMs can generate highly convincing, nuanced text, which can be misused for crafting elaborate phishing emails or social engineering schemes. These AI-generated messages are often indistinguishable from genuine communications, increasing the success rate of scams targeting individuals or organizations.

The refinement in language models poses a substantial risk, as it enables large-scale, targeted phishing campaigns. Addressing this threat requires enhanced vigilance, user education, and advanced detection systems, tailored to recognize subtle cues of deception in AI-generated content.

Malware Generation

Threat actors could leverage tools like ChatGPT to refine or create new malware, using the AI’s capabilities to generate and refactor code. By inputting descriptions of desired malicious functionalities, attackers can potentially receive code snippets or entire malware payloads. 

Generative AI tools lower the barrier to entry for less technically skilled criminals and result in a more complex threat landscape. Additionally, attackers could automate modification of malware code to make it more resilient detection, as it can rapidly evolve to bypass security measures.

Mitigation Strategies: How to Use ChatGPT Securely in the Enterprise

Here are some useful strategies for protecting against the security risks posed by large language models like ChatGPT.

Fact-Check All Generated Content

Employing an LLM like ChatGPT necessitates verifying the accuracy of generated content. Misinformation can propagate quickly, leading to decisions based on false premises. Establishing fact-checking protocols ensures the reliability of AI-generated outputs, safeguarding against the spread of inaccuracies.

Organizations should incorporate cross-verification steps and consult authoritative sources to validate generated content. Such measures enhance the credibility and usefulness of LLM applications in business settings.

Anonymize Data 

Data anonymization protects sensitive information processed by LLMs. Removing or disguising identifiable details prevents data breaches and maintains privacy. Implementing anonymization techniques is essential when feeding real-world data into LLMs for training or query responses.

Techniques like data masking, tokenization, and differential privacy can effectively anonymize data, minimizing the risk of exposure while retaining the utility of the information for AI applications.

Note: OpenAI recently introduced Team and Enterprise tiers of the ChatGPT tool, in which they claim that private data is not used to train their models. However, careful due diligence is required to confirm the details of these claims, and safeguards at the organizational level are still recommended.

Be Careful with APIs and Third-Party Apps 

Incorporating third-party applications and APIs into LLM integrations introduces additional security vulnerabilities. Rigorous vetting of external tools and strict access controls are crucial in safeguarding against unauthorized access and data leaks. In high-security environments, it might be advisable to ban the use of third-party plugins or custom GPTs in ChatGPT.

Organizations should conduct thorough security assessments of third-party solutions, ensuring they adhere to best practices and compliance standards. Maintaining tight control over API usage and monitoring for abnormal activities also helps mitigate risks associated with external integrations.

Don’t Overlook Traditional Security Tools 

While LLMs pose new security challenges, traditional cybersecurity tools remain vital. Antivirus software, firewalls, and Security Information and Event Management (SIEM) systems still play crucial roles in protecting against unauthorized access, injection attacks, and other suspicious activities related to LLM usage.

Integrating these tools into a comprehensive security strategy can extend protection to cover the emerging threats associated with generative AI technologies. As with all business-critical applications, continuous monitoring is important for identifying security incidents and gaining visibility over the threat landscape.

Exabeam Fusion: The Leading AI-Powered SIEM

Exabeam offers an AI-powered experience across the entire TDIR workflow. A combination of more than 1,800 pattern-matching rules and ML-based behavior models automatically detect potential security threats such as credential-based attacks, insider threats, and ransomware activity by identifying high risk user and entity activity. The industry-leading user and entity behavior analytics (UEBA) baselines normal activity for all users and entities, presenting all notable events chronologically.

Smart Timelines highlight the risk associated with each event, saving an analyst from writing hundreds of queries. Machine learning automates the alert triage workflow, adding UEBA context to dynamically identify, prioritize, and escalate alerts requiring the most attention.

The Exabeam platform can orchestrate and automate repeated workflows to over 100 third-party products with actions and operations, from semi- to fully automated activity. And Exabeam Outcomes Navigator maps the sources of the feeds that come into Exabeam products against the most common security use cases and suggests ways to improve coverage.