mlsanityData to 7 Oct 2026

From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy

arXiv 2307.00691 ↗First posted 3 Jul 2023cs.CR cs.AIcited by 137

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Undoubtedly, the evolution of Generative AI (GenAI) models has been the highlight of digital transformation in the year 2022. As the different GenAI models like ChatGPT and Google Bard continue to foster their complexity and capability, it's critical to understand its consequences from a cybersecurity perspective. Several instances recently have demonstrated the use of GenAI tools in both the defensive and offensive side of cybersecurity, and focusing on the social, ethical and privacy implications this technology possesses. This research paper highlights the limitations, challenges, potential risks, and opportunities of GenAI in the domain of cybersecurity and privacy. The work presents the vulnerabilities of ChatGPT, which can be exploited by malicious users to exfiltrate malicious information bypassing the ethical constraints on the model. This paper demonstrates successful example attacks like Jailbreaks, reverse psychology, and prompt injection attacks on the ChatGPT. The paper also investigates how cyber offenders can use the GenAI tools in developing cyber attacks, and explore the scenarios where ChatGPT can be used by adversaries to create social engineering attacks, phishing attacks, automated hacking, attack payload generation, malware creation, and polymorphic malware. This paper then examines defense techniques and uses GenAI tools to improve security measures, including cyber defense automation, reporting, threat intelligence, secure code generation and detection, attack identification, developing ethical guidelines, incidence response plans, and malware detection. We will also discuss the social, legal, and ethical implications of ChatGPT. In conclusion, the paper highlights open challenges and future directions to make this GenAI secure, safe, trustworthy, and ethical as the community understands its cybersecurity impacts.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 5 of its 56 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

5 of 56 references.

References · 5

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  5. 2021A Survey on Adversarial Attacks for Malware Analysiscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 63; 2025: 47; 2026: 10.

137 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  5. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  6. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  7. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  8. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  9. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  10. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  11. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  12. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
38 more
  1. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  2. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Modelscited by 56
  3. 2023GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Modelscited by 40
  4. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agentcited by 39
  5. 2023Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformerscited by 35
  6. 2024AI Hallucinations: A Misnomer Worth Clarifyingcited by 34
  7. 2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 28
  8. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 26
  9. 2023LLMs as Hackers: Autonomous Linux Privilege Escalation Attackscited by 25
  10. 2024Malla: Demystifying Real-world Large Language Model Integrated Malicious Servicescited by 24
  11. 2023Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questionscited by 23
  12. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  13. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 22
  14. 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
  15. 2023RatGPT: Turning online LLMs into Proxies for Malware Attackscited by 17
  16. 2024Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Reviewcited by 17
  17. 2023Negotiating with LLMS: Prompt Hacks, Skill Gaps, and Reasoning Deficitscited by 16
  18. 2025Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineeringcited by 15
  19. 2024Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awarenesscited by 14
  20. 2024Toxicity Detection for Freecited by 14
  21. 2024Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Modelscited by 14
  22. 2023From Chatbots to PhishBots? -- Preventing Phishing scams created using ChatGPT, Google Bard and Claudecited by 12
  23. 2024Quantized Delta Weight Is Safety Keepercited by 12
  24. 2024Risks and Opportunities of Open-Source Generative AIcited by 11
  25. 2024Efficient Detection of Toxic Prompts in Large Language Modelscited by 11
  26. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  27. 2023AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Modelscited by 10
  28. 2023Cyber Sentinel: Exploring Conversational Agents in Streamlining Security Tasks with GPT-4cited by 9
  29. 2024Natural Language Processing RELIES on Linguisticscited by 9
  30. 2024Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Surveycited by 9
  31. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  32. 2024Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intentcited by 8
  33. 2024Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Surveycited by 8
  34. 2025On the Impact of Noise in Differentially Private Text Rewritingcited by 8
  35. 2024AbuseGPT: Abuse of Generative AI ChatBots to Create Smishing Campaignscited by 7
  36. 2024From COBIT to ISO 42001: Evaluating Cybersecurity Frameworks for Opportunities, Risks, and Regulatory Compliance in Commercializing Large Language Modelscited by 7
  37. 2024A Survey on Human-AI Collaboration with Large Foundation Modelscited by 7
  38. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7

50 most cited of 137 All 137 (JSON)