mlsanityData to 14 Oct 2026

Improved Techniques for Optimization-Based Jailbreaking on Large Language Models

arXiv 2405.21018 ↗First posted 31 May 2024cs.LG cs.CL cs.CRcited by 110

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) are being rapidly developed, and a key component of their widespread deployment is their safety-related alignment. Many red-teaming efforts aim to jailbreak LLMs, where among these efforts, the Greedy Coordinate Gradient (GCG) attack's success has led to a growing interest in the study of optimization-based jailbreaking techniques. Although GCG is a significant milestone, its attacking efficiency remains unsatisfactory. In this paper, we present several improved (empirical) techniques for optimization-based jailbreaks like GCG. We first observe that the single target template of "Sure" largely limits the attacking performance of GCG; given this, we propose to apply diverse target templates containing harmful self-suggestion and/or guidance to mislead LLMs. Besides, from the optimization aspects, we propose an automatic multi-coordinate updating strategy in GCG (i.e., adaptively deciding how many tokens to replace in each step) to accelerate convergence, as well as tricks like easy-to-hard initialisation. Then, we combine these improved technologies to develop an efficient jailbreak method, dubbed I-GCG. In our experiments, we evaluate on a series of benchmarks (such as NeurIPS 2023 Red Teaming Track). The results demonstrate that our improved techniques can help GCG outperform state-of-the-art jailbreaking attacks and achieve nearly 100% attack success rate. The code is released at https://github.com/jiaxiaojunQAQ/I-GCG.

Key facts. Cited by 110 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 43 of its 54 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

43 of 54 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024Boosting Jailbreak Attack with Momentum2 May
  2. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  3. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  4. 2024Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?4 Apr
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  6. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models18 Mar
  7. 2024Distract Large Language Models for Automatic Jailbreak Attack13 Mar
  8. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling2 Mar
  9. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  10. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  11. 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
  12. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
12 more
  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  3. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  5. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  6. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  8. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  9. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  11. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  12. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2023Mistral 7Bcited by 3,164
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  8. 2022Red Teaming Language Models with Language Modelscited by 1,026
  9. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  10. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  11. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  12. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
7 more
  1. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  2. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  3. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 44
  4. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimizationcited by 32
  5. 2024Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Modelscited by 17
  6. 2024On the Multi-modal Vulnerability of Diffusion Modelscited by 15
  7. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 57; 2026: 40.

110 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  4. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  5. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  6. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 63
  7. 2024Large Language Model Safety: A Holistic Surveycited by 59
  8. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  9. 2024Perception-guided Jailbreak against Text-to-Image Modelscited by 40
  10. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 33
  11. 2025Adversarial Reasoning at Jailbreaking Timecited by 29
  12. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 26
38 more
  1. 2024Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Modelscited by 25
  2. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  3. 2026SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agentscited by 25
  4. 2025To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Modelscited by 22
  5. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 21
  6. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  7. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 19
  8. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  9. 2025LLM-Safety Evaluations Lack Robustnesscited by 18
  10. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 18
  11. 2025Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 16
  12. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  13. 2025Lifelong Safety Alignment for Language Modelscited by 15
  14. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  15. 2025Safety Alignment via Constrained Knowledge Unlearningcited by 14
  16. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  17. 2025SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAGcited by 13
  18. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 13
  19. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  20. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Searchcited by 11
  21. 2025Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructionscited by 11
  22. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 11
  23. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 10
  24. 2025Universal Jailbreak Suffixes Are Strong Attention Hijackerscited by 10
  25. 2025Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerabilitycited by 10
  26. 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Modelscited by 10
  27. 2025Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMscited by 9
  28. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  29. 2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
  30. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  31. 2025Towards the Worst-case Robustness of Large Language Modelscited by 7
  32. 2025Shh, don't say that! Domain Certification in LLMscited by 7
  33. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrievalcited by 7
  34. 2026Knowledge-Driven Multi-Turn Jailbreaking on Large Language Modelscited by 7
  35. 2025EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructionscited by 6
  36. 2025Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?cited by 6
  37. 2025AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Promptscited by 6
  38. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projectioncited by 6

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Models
  2. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation
  3. 2024Boosting Jailbreak Transferability for Large Language Models
  4. 2024Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization
  5. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  6. 2025Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses
  7. 2025CCJA: Context-Coherent Jailbreak Attack for Aligned Large Language Models
  8. 2026Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models