# From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy arXiv: https://arxiv.org/abs/2307.00691 ยท HTML: https://mlsanity.com/paper/2307.00691/ ## Key facts - Cited by 143 papers in the map (data through 14 Oct 2026). - Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. - Builds on 5 of its 56 references that are in the map. - First posted on arXiv 3 Jul 2023. ## Placed in - LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/) ## Builds on (most cited first, 5 of 5) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022) - [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020) - [A Survey on Adversarial Attacks for Malware Analysis](https://mlsanity.com/paper/2111.08223/) (2021) ## Built on by (most cited first, 30 of 143) - [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023) - [LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked](https://mlsanity.com/paper/2308.07308/) (2023) - [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024) - [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024) - [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024) - [Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey](https://mlsanity.com/paper/2402.09283/) (2024) - [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024) - [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024) - [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023) - [A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems](https://mlsanity.com/paper/2402.18649/) (2024) - [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024) - [JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models](https://mlsanity.com/paper/2407.01599/) (2024) - [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024) - [Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models](https://mlsanity.com/paper/2307.08487/) (2023) - [RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent](https://mlsanity.com/paper/2407.16667/) (2024) - [GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models](https://mlsanity.com/paper/2310.00737/) (2023) - [Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers](https://mlsanity.com/paper/2311.10242/) (2023) - [AI Hallucinations: A Misnomer Worth Clarifying](https://mlsanity.com/paper/2401.06796/) (2024) - [Malla: Demystifying Real-world Large Language Model Integrated Malicious Services](https://mlsanity.com/paper/2401.03315/) (2024) - [An Early Categorization of Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2402.00898/) (2024) - [LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks](https://mlsanity.com/paper/2310.11409/) (2023) - [Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMs](https://mlsanity.com/paper/2312.04782/) (2023) - [From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking](https://mlsanity.com/paper/2406.14859/) (2024) - [Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions](https://mlsanity.com/paper/2308.10443/) (2023) - [Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda](https://mlsanity.com/paper/2404.09554/) (2024) - [Mapping the Ethics of Generative AI: A Comprehensive Scoping Review](https://mlsanity.com/paper/2402.08323/) (2024) - [RatGPT: Turning online LLMs into Proxies for Malware Attacks](https://mlsanity.com/paper/2308.09183/) (2023) - [Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review](https://mlsanity.com/paper/2401.01519/) (2024) - [Negotiating with LLMS: Prompt Hacks, Skill Gaps, and Reasoning Deficits](https://mlsanity.com/paper/2312.03720/) (2023) - [Toxicity Detection for Free](https://mlsanity.com/paper/2405.18822/) (2024) ## Abstract (arXiv) Undoubtedly, the evolution of Generative AI (GenAI) models has been the highlight of digital transformation in the year 2022. As the different GenAI models like ChatGPT and Google Bard continue to foster their complexity and capability, it's critical to understand its consequences from a cybersecurity perspective. Several instances recently have demonstrated the use of GenAI tools in both the defensive and offensive side of cybersecurity, and focusing on the social, ethical and privacy implications this technology possesses. This research paper highlights the limitations, challenges, potential risks, and opportunities of GenAI in the domain of cybersecurity and privacy. The work presents the vulnerabilities of ChatGPT, which can be exploited by malicious users to exfiltrate malicious information bypassing the ethical constraints on the model. This paper demonstrates successful example attacks like Jailbreaks, reverse psychology, and prompt injection attacks on the ChatGPT. The paper also investigates how cyber offenders can use the GenAI tools in developing cyber attacks, and explore the scenarios where ChatGPT can be used by adversaries to create social engineering attacks, phishing attacks, automated hacking, attack payload generation, malware creation, and polymorphic malware. This paper then examines defense techniques and uses GenAI tools to improve security measures, including cyber defense automation, reporting, threat intelligence, secure code generation and detection, attack identification, developing ethical guidelines, incidence response plans, and malware detection. We will also discuss the social, legal, and ethical implications of ChatGPT. In conclusion, the paper highlights open challenges and future directions to make this GenAI secure, safe, trustworthy, and ethical as the community understands its cybersecurity impacts.