mlsanityData to 7 Oct 2026

MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

arXiv 2311.07689 ↗First posted 13 Nov 2023cs.CLcited by 161

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Red-teaming is a common practice for mitigating unsafe behaviors in Large Language Models (LLMs), which involves thoroughly assessing LLMs to identify potential flaws and addressing them with responsible and accurate responses. While effective, manual red-teaming is costly, and existing automatic red-teaming typically discovers safety risks without addressing them. In this paper, we propose a Multi-round Automatic Red-Teaming (MART) method, which incorporates both automatic adversarial prompt writing and safe response generation, significantly increasing red-teaming scalability and the safety of the target LLM. Specifically, an adversarial LLM and a target LLM interplay with each other in an iterative manner, where the adversarial LLM aims to generate challenging prompts that elicit unsafe responses from the target LLM, while the target LLM is fine-tuned with safety aligned data on these adversarial prompts. In each round, the adversarial LLM crafts better attacks on the updated target LLM, while the target LLM also improves itself through safety fine-tuning. On adversarial prompt benchmarks, the violation rate of an LLM with limited safety alignment reduces up to 84.7% after 4 rounds of MART, achieving comparable performance to LLMs with extensive adversarial prompt writing. Notably, model helpfulness on non-adversarial prompts remains stable throughout iterations, indicating the target LLM maintains strong performance on instruction following.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 18 of its 25 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

18 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  2. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 16

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  8. 2023LIMA: Less Is More for Alignmentcited by 1,080
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  12. 2020Aligning AI With Shared Human Valuescited by 706
4 more
  1. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  2. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  3. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  4. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 56; 2025: 61; 2026: 41.

161 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  5. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  6. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  9. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  10. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  11. 2024Towards Conversational Diagnostic AIcited by 79
  12. 2024Safeguarding Large Language Models: A Surveycited by 73
38 more
  1. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  2. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  3. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  4. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 63
  5. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  6. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 57
  7. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 50
  8. 2023Bypassing the Safety Training of Open-Source LLMs with Priming Attackscited by 48
  9. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  10. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  11. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  12. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  13. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
  14. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  15. 2024ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Userscited by 32
  16. 2024Self-Supervised Visual Preference Alignmentcited by 31
  17. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 31
  18. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templatescited by 29
  19. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  20. 2024When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Cipherscited by 27
  21. 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 27
  22. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  23. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  24. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  25. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  26. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  27. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  28. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Modelscited by 22
  29. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  30. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  31. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  32. 2024SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Modelscited by 16
  33. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  34. 2025Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Controlcited by 14
  35. 2024Toward Optimal LLM Alignments Using Two-Player Gamescited by 13
  36. 2024Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Modelscited by 13
  37. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 13
  38. 2024Automated Progressive Red Teamingcited by 12

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs
  2. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
  3. 2025RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
  4. 2024Gradient-Based Language Model Red Teaming
  5. 2025Multi-lingual Multi-turn Automated Red Teaming for LLMs
  6. 2025LLM-Safety Evaluations Lack Robustness
  7. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Tester
  8. 2024Large Language Model Safety: A Holistic Survey