# Beyond the Safeguards: Exploring the Security Risks of ChatGPT

arXiv: https://arxiv.org/abs/2305.08005 · HTML: https://mlsanity.com/paper/2305.08005/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 13 May 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2102.02503/) (2021)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [Detoxifying Language Models Risks Marginalizing Minority Voices](https://mlsanity.com/paper/2104.06390/) (2021)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT](https://mlsanity.com/paper/2304.12298/) (2023)
- [Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?](https://mlsanity.com/paper/1904.05506/) (2019)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Mitigating harm in language models with conditional-likelihood filtration](https://mlsanity.com/paper/2108.07790/) (2021)
- [Generating Fake Cyber Threat Intelligence Using Transformer-Based Models](https://mlsanity.com/paper/2102.04351/) (2021)
- [Targeted Phishing Campaigns using Large Scale Language Models](https://mlsanity.com/paper/2301.00665/) (2022)
- [Adaptive Sampling Strategies to Construct Equitable Training Datasets](https://mlsanity.com/paper/2202.01327/) (2022)
- [Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2303.02891/) (2023)
- [Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements](https://mlsanity.com/paper/2302.09270/) (2023)

## Built on by (most cited first, 30 of 36)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction](https://mlsanity.com/paper/2402.18104/) (2024)
- [The Art of Saying No: Contextual Noncompliance in Language Models](https://mlsanity.com/paper/2407.12043/) (2024)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [A Complete Survey on LLM-based AI Chatbots](https://mlsanity.com/paper/2406.16937/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [The Ethics of Interaction: Mitigating Security Threats in LLMs](https://mlsanity.com/paper/2401.12273/) (2024)
- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [Distilled GPT for Source Code Summarization](https://mlsanity.com/paper/2308.14731/) (2023)
- [Farsight: Fostering Responsible AI Awareness During AI Application Prototyping](https://mlsanity.com/paper/2402.15350/) (2024)
- [On the Detectability of ChatGPT Content: Benchmarking, Methodology, and Evaluation through the Lens of Academic Writing](https://mlsanity.com/paper/2306.05524/) (2023)
- [Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions](https://mlsanity.com/paper/2308.10443/) (2023)
- [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024)
- [Mapping the Ethics of Generative AI: A Comprehensive Scoping Review](https://mlsanity.com/paper/2402.08323/) (2024)
- [RatGPT: Turning online LLMs into Proxies for Malware Attacks](https://mlsanity.com/paper/2308.09183/) (2023)
- [Efficient Avoidance of Vulnerabilities in Auto-completed Smart Contract Code Using Vulnerability-constrained Decoding](https://mlsanity.com/paper/2309.09826/) (2023)
- [From Chatbots to PhishBots? -- Preventing Phishing scams created using ChatGPT, Google Bard and Claude](https://mlsanity.com/paper/2310.19181/) (2023)
- [SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile](https://mlsanity.com/paper/2404.13528/) (2024)
- [Risk and Response in Large Language Models: Evaluating Key Threat Categories](https://mlsanity.com/paper/2403.14988/) (2024)
- [Ask Again, Then Fail: Large Language Models' Vacillations in Judgment](https://mlsanity.com/paper/2310.02174/) (2023)
- [Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency](https://mlsanity.com/paper/2402.19366/) (2024)
- [Learning to Prompt in the Classroom to Understand AI Limits: A pilot study](https://mlsanity.com/paper/2307.01540/) (2023)
- [FreezeAsGuard: Mitigating Illegal Adaptation of Diffusion Models via Selective Tensor Freezing](https://mlsanity.com/paper/2405.17472/) (2024)
- [Unanswerability Evaluation for Retrieval Augmented Generation](https://mlsanity.com/paper/2412.12300/) (2024)
- [Tagengo: A Multilingual Chat Dataset](https://mlsanity.com/paper/2405.12612/) (2024)
- [Whispers in the Machine: Confidentiality in Agentic Systems](https://mlsanity.com/paper/2402.06922/) (2024)
- [SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://mlsanity.com/paper/2502.18468/) (2025)
- [Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessment](https://mlsanity.com/paper/2508.08629/) (2025)
- [Is On-Device AI Broken and Exploitable? Assessing the Trust and Ethics in Small Language Models](https://mlsanity.com/paper/2406.05364/) (2024)

## Abstract (arXiv)

The increasing popularity of large language models (LLMs) such as ChatGPT has led to growing concerns about their safety, security risks, and ethical implications. This paper aims to provide an overview of the different types of security risks associated with ChatGPT, including malicious text and code generation, private data disclosure, fraudulent services, information gathering, and producing unethical content. We present an empirical study examining the effectiveness of ChatGPT's content filters and explore potential ways to bypass these safeguards, demonstrating the ethical implications and security risks that persist in LLMs even when protections are in place. Based on a qualitative analysis of the security implications, we discuss potential strategies to mitigate these risks and inform researchers, policymakers, and industry professionals about the complex security challenges posed by LLMs like ChatGPT. This study contributes to the ongoing discussion on the ethical and security implications of LLMs, underscoring the need for continued research in this area.