mlsanityData to 7 Oct 2026

Gradient-Based Language Model Red Teaming

arXiv 2401.16656 ↗First posted 30 Jan 2024cs.CLcited by 33

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 26 of its 32 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

26 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  2. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimization27 May
  3. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  12. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
11 more
  1. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  2. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  3. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  4. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  5. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  6. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  7. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  8. 2018Unsupervised Text Style Transfer using Language Models as Discriminatorscited by 70
  9. 2021A Plug-and-Play Method for Controlled Text Generationcited by 55
  10. 2018Content preserving text generation with attribute controlscited by 47
  11. 2022Robust Conversational Agents against Imperceptible Toxicity Triggerscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 11; 2026: 4.

33 papers.

Sort:

By area:

  1. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  2. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  3. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  4. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  5. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  6. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  7. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  8. 2024ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddingscited by 31
  9. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 26
  10. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  11. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  12. 2024Holistic Safety and Responsibility Evaluations of Advanced AI Modelscited by 18
21 more
  1. 2024SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Modelscited by 16
  2. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  3. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  4. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  5. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  6. 2024DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreakcited by 8
  7. 2025Adversarial Preference Learning for Robust LLM Alignmentcited by 7
  8. 2024Securing Multi-turn Conversational Language Models From Distributed Backdoor Triggerscited by 5
  9. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  10. 2025Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraintscited by 4
  11. 2024Impact of Non-Standard Unicode Characters on Security and Comprehension in Large Language Modelscited by 3
  12. 2024Automated Adversarial Discovery for Safety Classifierscited by 3
  13. 2024ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Promptscited by 3
  14. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  15. 2025Generating Query-Relevant Document Summaries via Reinforcement Learningcited by 1
  16. 2025Reimagining Safety Alignment with An Imagecited by 1
  17. 2026FERRET: Framework for Expansion Reliant Red Teamingcited by 1
  18. 2025Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreakingcited by 0
  19. 2025Don't Walk the Line: Boundary Guidance for Filtered Generationcited by 0
  20. 2026SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimizationcited by 0
  21. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
  3. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Tester
  4. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
  5. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Models
  6. 2026Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming
  7. 2025Multi-lingual Multi-turn Automated Red Teaming for LLMs
  8. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment