mlsanityData to 7 Oct 2026

Coercing LLMs to do and reveal (almost) anything

arXiv 2402.14020 ↗First posted 21 Feb 2024cs.LG cs.CL cs.CRcited by 88

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

It has recently been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements. In this work, we argue that the spectrum of adversarial attacks on LLMs is much larger than merely jailbreaking. We provide a broad overview of possible attack surfaces and attack goals. Based on a series of concrete examples, we discuss, categorize and systematize attacks that coerce varied unintended behaviors, such as misdirection, model control, denial-of-service, or data extraction. We analyze these attacks in controlled experiments, and find that many of them stem from the practice of pre-training LLMs with coding capabilities, as well as the continued existence of strange "glitch" tokens in common LLM vocabularies that should be removed for security reasons.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 53 of its 72 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

53 of 72 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  2. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning19 Jan
  3. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
  4. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective12 Jan
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  9. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  10. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  11. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  12. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
3 more
  1. 2023Detecting Language Model Attacks with Perplexity27 Aug
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
26 more
  1. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  2. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  5. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  6. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  8. 2023Are aligned neural networks adversarially aligned?cited by 305
  9. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  10. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  11. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  12. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  13. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  14. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  15. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  16. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  17. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  18. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  19. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  20. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  21. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  22. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  23. 2023Language Model Inversioncited by 66
  24. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
  25. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  26. 2023LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?cited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 35; 2026: 13.

88 papers.

Sort:

By area:

  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  3. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  4. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  7. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  8. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  9. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  10. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  11. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  12. 2025How much do language models memorize?cited by 53
38 more
  1. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  2. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 44
  3. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  4. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  5. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  6. 2024Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultracited by 36
  7. 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 34
  8. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  9. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 33
  10. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 30
  11. 2024Compromising Embodied Agents with Contextual Backdoor Attackscited by 29
  12. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  13. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  14. 2024Prompt Leakage effect and defense strategies for multi-turn LLM interactionscited by 23
  15. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  16. 2025ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinkingcited by 22
  17. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  18. 2024Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You Incited by 21
  19. 2025MCPXKIT: The Unified Toolkit for Analyzing Model Context Protocol Securitycited by 21
  20. 2024Bayesian scaling laws for in-context learningcited by 18
  21. 2025CORBA: Contagious Recursive Blocking Attacks on Multi-Agent Systems Based on Large Language Modelscited by 18
  22. 2026Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agentscited by 14
  23. 2026Clawdrain: Exploiting Tool-Calling Chains for Stealthy Token Exhaustion in OpenClaw Agentscited by 14
  24. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  25. 2024Black-Box Adversarial Attacks on LLM-Based Code Completioncited by 13
  26. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  27. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  28. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  29. 2025LingoLoop Attack: Trapping MLLMs via Linguistic Context and State Entrapment into Endless Loopscited by 9
  30. 2025Fast and Expressive Multi-Byte Prediction with Probabilistic Circuitscited by 9
  31. 2024Problematic Tokens: Tokenizer Bias in Large Language Modelscited by 8
  32. 2025A Survey of Theory of Mind in Large Language Models: Evaluations, Representations, and Safety Riskscited by 8
  33. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  34. 2024Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizerscited by 7
  35. 2025Improving Large Language Model Safety with Contrastive Representation Learningcited by 7
  36. 2025LoopLLM: Transferable Energy-Latency Attacks in LLMs via Repetitive Generationcited by 7
  37. 2025The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)cited by 6
  38. 2025BadReasoner: Planting Tunable Overthinking Backdoors into Large Reasoning Models for Fun or Profitcited by 6

50 most cited of 88 All 88 (JSON)