mlsanityData to 7 Oct 2026

COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability

arXiv 2402.08679 ↗First posted 13 Feb 2024cs.LG cs.AI cs.CLcited by 162

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreaks on large language models (LLMs) have recently received increasing attention. For a comprehensive assessment of LLM safety, it is essential to consider jailbreaks with diverse attributes, such as contextual coherence and sentiment/stylistic variations, and hence it is beneficial to study controllable jailbreaking, i.e. how to enforce control on LLM attacks. In this paper, we formally formulate the controllable attack generation problem, and build a novel connection between this problem and controllable text generation, a well-explored topic of natural language processing. Based on this connection, we adapt the Energy-based Constrained Decoding with Langevin Dynamics (COLD), a state-of-the-art, highly efficient algorithm in controllable text generation, and introduce the COLD-Attack framework which unifies and automates the search of adversarial LLM attacks under a variety of control requirements such as fluency, stealthiness, sentiment, and left-right-coherence. The controllability enabled by COLD-Attack leads to diverse new jailbreak scenarios which not only cover the standard setting of generating fluent (suffix) attack with continuation constraint, but also allow us to address new controllable attack settings such as revising a user query adversarially with paraphrasing constraint, and inserting stealthy attacks in context with position constraint. Our extensive experiments on various LLMs (Llama-2, Mistral, Vicuna, Guanaco, GPT-3.5, and GPT-4) show COLD-Attack's broad applicability, strong controllability, high success rate, and attack transferability. Our code is available at https://github.com/Yu-Fangxu/COLD-Attack.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 78 of its 94 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

78 of 94 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models17 Jan
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  3. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak7 Dec
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  5. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  6. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  9. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  10. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  11. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
3 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  3. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  12. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
38 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  3. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  11. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  12. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  13. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  14. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  15. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  16. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  17. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  18. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  19. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  20. 2023Are aligned neural networks adversarially aligned?cited by 305
  21. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  22. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  23. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  24. 2023Pretraining Language Models with Human Preferencescited by 245
  25. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  26. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  27. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  28. 2018Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translationcited by 181
  29. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  30. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 157
  31. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  32. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  33. 2023Controlled Decoding from Language Modelscited by 133
  34. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  35. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  36. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  37. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  38. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 49; 2025: 78; 2026: 35.

162 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  7. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  8. 2024Safeguarding Large Language Models: A Surveycited by 73
  9. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  10. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 66
  11. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 57
  12. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 50
38 more
  1. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  2. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  3. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  4. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 44
  5. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  6. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agentcited by 39
  7. 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaoscited by 39
  8. 2024Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultracited by 36
  9. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  10. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 33
  11. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  12. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 29
  13. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  14. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialoguecited by 28
  15. 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 27
  16. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  17. 2025Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generationcited by 24
  18. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  19. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  20. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  21. 2025ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinkingcited by 22
  22. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  23. 2024RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Evictioncited by 21
  24. 2024Endless Jailbreaks with Bijection Learningcited by 18
  25. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 16
  26. 2024Adversarial Decoding: Generating Readable Documents for Adversarial Objectivescited by 16
  27. 2024Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilitiescited by 16
  28. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  29. 2024MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usabilitycited by 14
  30. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  31. 2024Legilimens: Practical and Unified Content Moderation for Large Language Model Servicescited by 13
  32. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  33. 2025Chain-of-Thought Hijackingcited by 13
  34. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  35. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizerscited by 12
  36. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  37. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  38. 2025JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoringcited by 12

50 most cited of 162 All 162 (JSON)