mlsanityData to 7 Oct 2026

Boosting Jailbreak Attack with Momentum

arXiv 2405.01229 ↗First posted 2 May 2024cs.LG cs.AI cs.CLcited by 38

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable success across diverse tasks, yet they remain vulnerable to adversarial attacks, notably the well-known jailbreak attack. In particular, the Greedy Coordinate Gradient (GCG) attack has demonstrated efficacy in exploiting this vulnerability by optimizing adversarial prompts through a combination of gradient heuristics and greedy search. However, the efficiency of this attack has become a bottleneck in the attacking process. To mitigate this limitation, in this paper we rethink the generation of the adversarial prompts through an optimization lens, aiming to stabilize the optimization process and harness more heuristic insights from previous optimization iterations. Specifically, we propose the Momentum Accelerated GCG (MAC) attack, which integrates a momentum term into the gradient heuristic to boost and stabilize the random search for tokens in adversarial prompts. Experimental results showcase the notable enhancement achieved by MAC over baselines in terms of attack success rate and optimization efficiency. Moreover, we demonstrate that MAC can still exhibit superior performance for transfer attacks and models under defense mechanisms. Our code is available at https://github.com/weizeming/momentum-attack-llm.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 38 of its 58 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

38 of 58 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  2. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  4. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking16 Nov
  5. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  7. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  8. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  9. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  10. 2023Detecting Language Model Attacks with Perplexity27 Aug
  11. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
2 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  2. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  7. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  10. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  11. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  12. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
12 more
  1. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  2. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  3. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  4. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  5. 2023Combating Misinformation in the Age of LLMs: Opportunities and Challengescited by 94
  6. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  7. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 88
  8. 2023Robust Classification via a Single Diffusion Modelcited by 79
  9. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  10. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  11. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21
  12. 2023Weighted Automata Extraction and Explanation of Recurrent Neural Networks for Natural Language Taskscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 20; 2026: 13.

38 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  3. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  4. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  5. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 20
  6. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  7. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  8. 2024Efficient Universal Goal Hijacking with Semantics-guided Prompt Organizationcited by 13
  9. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  10. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  11. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  12. 2025Towards the Worst-case Robustness of Large Language Modelscited by 7
26 more
  1. 2025AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Modelscited by 7
  2. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrievalcited by 7
  3. 2025Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representationscited by 7
  4. 2025Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?cited by 6
  5. 2026Knowledge-Driven Multi-Turn Jailbreaking on Large Language Modelscited by 6
  6. 2025Geneshift: Impact of different scenario shift on Jailbreaking LLMcited by 4
  7. 2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
  8. 2025When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputscited by 3
  9. 2025Imperceptible Jailbreaking against Large Language Modelscited by 3
  10. 2025BitBypass: A New Direction in Jailbreaking Aligned Large Language Models with Bitstream Camouflagecited by 2
  11. 2025NonTextual Target Attackcited by 2
  12. 2025EquaCode: A Multi-Strategy Jailbreak Approach for Large Language Models via Equation Solving and Code Completioncited by 2
  13. 2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
  14. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  15. 2025ReGA: Model-Based Safeguard for LLMs via Representation-Guided Abstractioncited by 1
  16. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  17. 2025Differences That Matter: Auditing Models for Capability Gap Discovery and Rectificationcited by 1
  18. 2026Break Me If You Can: Self-Jailbreaking of Aligned LLMs via Lexical Insertion Promptingcited by 1
  19. 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Modelscited by 1
  20. 2026The Salami Slicing Threat: Exploiting Cumulative Risks in LLM Systemscited by 1
  21. 2026SlotGCG: Exploiting the Positional Vulnerability in LLMs for Jailbreak Attackscited by 1
  22. 2026FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teamingcited by 1
  23. 2026GradingAttack: Exposing Security Vulnerabilities in LLM Based Educational Grading Agentscited by 0
  24. 2026Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMscited by 0
  25. 2026Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversioncited by 0
  26. 2026Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Codecited by 0