mlsanityData to 14 Oct 2026

A Red Teaming Roadmap Towards System-Level Safety

arXiv 2506.05376 ↗First posted 30 May 2025cs.CR cs.AIcited by 2

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Model (LLM) safeguards, which implement request refusals, have become a widely adopted mitigation strategy against misuse. At the intersection of adversarial machine learning and AI safety, safeguard red teaming has effectively identified critical vulnerabilities in state-of-the-art refusal-trained LLMs. However, in our view the many conference submissions on LLM red teaming do not, in aggregate, prioritize the right research problems. First, testing against clear product safety specifications should take a higher priority than abstract social biases or ethical principles. Second, red teaming should prioritize realistic threat models that represent the expanding risk landscape and what real attackers might do. Finally, we contend that system-level safety is a necessary step to move red teaming research forward, as AI models present new threats as well as affordances for threat mitigation (e.g., detection and banning of malicious users) once placed in a deployment context. Adopting these priorities will be necessary in order for red teaming research to adequately address the slate of new threats that rapid AI advances present today and will present in the very near future.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 79 of its 100 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

79 of 100 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks20 May
  2. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents9 May
  3. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models7 Mar
  4. 2023AI Control: Improving Safety Despite Intentional Subversion12 Dec

Other references · 75

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  8. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  12. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
38 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  4. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  7. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  8. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  9. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  12. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  13. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  14. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  15. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  16. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  17. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  18. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  19. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  20. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
  21. 2023An Overview of Catastrophic AI Riskscited by 222
  22. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 208
  23. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
  24. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  25. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  26. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  27. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 145
  28. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
  29. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  30. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 116
  31. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  32. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 99
  33. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  34. 2024Beyond Browsing: API-Based Web Agentscited by 78
  35. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 72
  36. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  37. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  38. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 65

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreakingcited by 0
  2. 2026LLJ Cards: Best practices for the Use of LLMs as Judgescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models
  2. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)
  3. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
  5. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
  6. 2025The Automation Advantage in AI Red Teaming
  7. 2025Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
  8. 2026CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generation