mlsanityData to 13 Oct 2026

REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective

arXiv 2502.17254 ↗First posted 24 Feb 2025cs.LGcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

To circumvent the alignment of large language models (LLMs), current optimization-based adversarial attacks usually craft adversarial prompts by maximizing the likelihood of a so-called affirmative response. An affirmative response is a manually designed start of a harmful answer to an inappropriate request. While it is often easy to craft prompts that yield a substantial likelihood for the affirmative response, the attacked model frequently does not complete the response in a harmful manner. Moreover, the affirmative objective is usually not adapted to model-specific preferences and essentially ignores the fact that LLMs output a distribution over responses. If low attack success under such an objective is taken as a measure of robustness, the true robustness might be grossly overestimated. To alleviate these flaws, we propose an adaptive and semantic optimization problem over the population of responses. We derive a generally applicable objective via the REINFORCE policy-gradient formalism and demonstrate its efficacy with the state-of-the-art jailbreak algorithms Greedy Coordinate Gradient (GCG) and Projected Gradient Descent (PGD). For example, our objective doubles the attack success rate (ASR) on Llama3 and increases the ASR from 2% to 50% with circuit breaker defense.

Key facts. Cited by 15 papers in the map (data through 13 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 57 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

43 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs13 Jun
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  4. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 38

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,215
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,405
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,404
  6. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,265
  7. 2022Red Teaming Language Models with Language Modelscited by 1,014
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,009
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 718
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 664
  11. 2020On Adaptive Attacks to Adversarial Example Defensescited by 554
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 485
26 more
  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 483
  2. 2024A StrongREJECT for Empty Jailbreakscited by 379
  3. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 335
  4. 2024Improving Alignment and Robustness with Circuit Breakerscited by 318
  5. 2019Adversarial Attacks and Defenses in Images, Graphs and Text: A Reviewcited by 236
  6. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  7. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 173
  8. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 171
  9. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  10. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 107
  11. 2019SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editingcited by 93
  12. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 87
  13. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  14. 2023Are Defenses for Graph Neural Networks Robust?cited by 38
  15. 2021Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustnesscited by 35
  16. 2024Gradient-Based Language Model Red Teamingcited by 34
  17. 2024InfAlign: Inference-aware language model alignmentcited by 34
  18. 2023Adversarial Training for Graph Neural Networks: Pitfalls, Solutions, and New Directionscited by 28
  19. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  20. 2022TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimizationcited by 21
  21. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  22. 2023Semantic Adversarial Attacks via Diffusion Modelscited by 12
  23. 2024Revisiting the Robust Alignment of Circuit Breakerscited by 11
  24. 2023Assessing Robustness via Score-Based Adversarial Image Generationcited by 6
  25. 2024LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMscited by 5
  26. 2024Adversarial Robustness of Graph Transformerscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 123
  2. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  3. 2025Improving Large Language Model Safety with Contrastive Representation Learningcited by 7
  4. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  5. 2025Mind the Web: The Security of Web Use Agentscited by 5
  6. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  7. 2025Injecting Falsehoods: Adversarial Man-in-the-Middle Attacks Undermining Factual Recall in LLMscited by 1
  8. 2026One Word is Enough: Minimal Adversarial Perturbations for Neural Text Rankingcited by 0
  9. 2026Closing the Distribution Gap in Adversarial Training for LLMscited by 0
  10. 2026Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testingcited by 0
  11. 2026Test-Time Training Undermines Safety Guardrailscited by 0
  12. 2026All you need to break LLMs are Black-Box, Adapting, Efficient, Transferable, Harmful, Applicable ... Attackscited by 0
3 more
  1. 2026Eliciting Intrinsic Hallucinations in LLMs via Semantically Equivalent Adversarial Attackscited by 0
  2. 2026Truth Lies Deep: Countering Semantic Camouflage via Latent Intent Verificationcited by 0
  3. 2026From Expected Harmfulness to Likelihood: A Probabilistic Reformulation of Jailbreaking LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sampling-aware Adversarial Attacks Against Large Language Models
  2. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxation
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models
  4. 2026Reward Stealing Attack on Large Language Models
  5. 2024Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?
  6. 2026Adversarial Reframing: A Framework for Targeted Generation in Language Models
  7. 2025The Resurgence of GCG Adversarial Attacks on Large Language Models
  8. 2025Safeguarding Efficacy in Large Language Models: Evaluating Resistance to Human-Written and Algorithmic Adversarial Prompts