mlsanityData to 14 Oct 2026

Training a General Purpose Automated Red Teaming Model

arXiv 2604.23067 ↗First posted 24 Apr 2026cs.CR cs.CLcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Automated methods for red teaming LLMs are an important tool to identify LLM vulnerabilities that may not be covered in static benchmarks, allowing for more thorough probing. They can also adapt to each specific LLM to discover weaknesses unique to it. Most current automated red teaming methods are intended for tackling safety and content moderation. Thus, they make use of content safety models as evaluators and optimize for circumventing them, and as such, have not been tested with other adversarial intents not typically captured by these. We propose a pipeline for training a red teaming model that can generalize to arbitrary adversarial goals, including objectives it has not been directly trained on, and that does not depend on the existence of a pre-existing evaluator available at training time. We demonstrate that finetuning small models, such as Qwen3-8B, using this pipeline results in a substantial improvement in their ability to generate attacks for both in and out of domain adversarial goals.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 23 of its 44 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

23 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Trust No AI: Prompt Injection Along The CIA Security Triad8 Dec
  2. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov

Other references · 21

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  7. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  8. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  9. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  10. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  11. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  12. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
9 more
  1. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 125
  2. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  3. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
  4. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  5. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 43
  6. 2024The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cited by 22
  7. 2025Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilitiescited by 15
  8. 2024Toxicity of the Commons: Curating Open-Source Pre-Training Datacited by 7
  9. 2023CONSCENDI: A Contrastive and Scenario-Guided Distillation Approach to Guardrail Models for Virtual Assistantscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming
  2. 2024Automated Progressive Red Teaming
  3. 2025SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning
  4. 2024Purple-teaming LLMs with Adversarial Defender Training
  5. 2026GPT-Red: Automated Red Teaming via Self-Play at Scale
  6. 2025Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
  7. 2026Generating Attacks for LLMs with GFlowNets
  8. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch