mlsanityData to 7 Oct 2026

SoK: Prompt Hacking of Large Language Models

arXiv 2410.13901 ↗First posted 16 Oct 2024cs.CR cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

The safety and robustness of large language models (LLMs) based applications remain critical challenges in artificial intelligence. Among the key threats to these applications are prompt hacking attacks, which can significantly undermine the security and reliability of LLM-based systems. In this work, we offer a comprehensive and systematic overview of three distinct types of prompt hacking: jailbreaking, leaking, and injection, addressing the nuances that differentiate them despite their overlapping characteristics. To enhance the evaluation of LLM-based applications, we propose a novel framework that categorizes LLM responses into five distinct classes, moving beyond the traditional binary classification. This approach provides more granular insights into the AI's behavior, improving diagnostic precision and enabling more targeted enhancements to the system's safety and robustness.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 29 of its 53 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

29 of 53 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  10. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  12. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
17 more
  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  3. 2023Pretraining Language Models with Human Preferencescited by 245
  4. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  6. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  7. 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacycited by 137
  8. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  9. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 75
  10. 2023Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Modelscited by 67
  11. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
  12. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  13. 2023Unveiling Security, Privacy, and Ethical Concerns of ChatGPTcited by 53
  14. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  15. 2023Knowledge Sanitization of Large Language Modelscited by 34
  16. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuningcited by 24
  17. 2023Jailbreaker in Jail: Moving Target Defense for Large Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2026Security Considerations for Artificial Intelligence Agentscited by 7
  2. 2024SoK: Blockchain-Based Decentralized AI (DeAI)cited by 6
  3. 2025Mind the Web: The Security of Web Use Agentscited by 5
  4. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  5. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1