mlsanityData to 7 Oct 2026

Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models

arXiv 2503.01742 ↗First posted 3 Mar 2025cs.CLcited by 11

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

The rapid growth of Large Language Models (LLMs) presents significant privacy, security, and ethical concerns. While much research has proposed methods for defending LLM systems against misuse by malicious actors, researchers have recently complemented these efforts with an offensive approach that involves red teaming, i.e., proactively attacking LLMs with the purpose of identifying their vulnerabilities. This paper provides a concise and practical overview of the LLM red teaming literature, structured so as to describe a multi-component system end-to-end. To motivate red teaming we survey the initial safety needs of some high-profile LLMs, and then dive into the different components of a red teaming system as well as software packages for implementing them. We cover various attack methods, strategies for attack-success evaluation, metrics for assessing experiment outcomes, as well as a host of other considerations. Our survey will be useful for any reader who wants to rapidly obtain a grasp of the major red teaming concepts for their own use in practical applications.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 70 of its 110 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

70 of 110 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System1 Oct
  2. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction25 Sep
  3. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI23 Sep
  4. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent23 Jul
  5. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)20 Jul
  6. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuning28 May
  7. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Models31 Mar
  8. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks6 Mar
  9. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents15 Feb
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  11. 2024Gradient-Based Language Model Red Teaming30 Jan
  12. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?29 Jan
7 more
  1. 2023AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications14 Nov
  2. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  3. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming10 Nov
  4. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  5. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
38 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  3. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  6. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  7. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  8. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  9. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  10. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  11. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  12. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 206
  13. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  14. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  15. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  16. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  17. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  18. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
  19. 2020MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient Estimationcited by 99
  20. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  21. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  22. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  23. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  24. 2020Reevaluating Adversarial Examples in Natural Languagecited by 83
  25. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 80
  26. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 79
  27. 2024Safeguarding Large Language Models: A Surveycited by 73
  28. 2024Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialoguecited by 64
  29. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoningcited by 62
  30. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  31. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  32. 2024Large Language Model Safety: A Holistic Surveycited by 56
  33. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 46
  34. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  35. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attackscited by 30
  36. 2024Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreakingcited by 30
  37. 2024Poisoned LangChain: Jailbreak LLMs by LangChaincited by 14
  38. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignmentcited by 14

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 7.

11 papers.

  1. 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competitioncited by 22
  2. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  3. 2026Mapping Human Anti-collusion Mechanisms to Multi-agent AI Systemscited by 4
  4. 2025DoubleAgents: Human-Agent Alignment in a Socially Embedded Workflowcited by 1
  5. 2026Agentic AI in Healthcare & Medicine: A Seven-Dimensional Taxonomy for Empirical Evaluation of LLM-based Agentscited by 1
  6. 2025The Laminar Flow Hypothesis: Detecting Jailbreaks via Semantic Turbulence in Large Language Modelscited by 0
  7. 2026Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemmacited by 0
  8. 2026A Systematic Investigation of RL-Jailbreaking in LLMscited by 0
  9. 2026Protecting patient privacy in clinical foundation models: Technical and legal perspectivescited by 0
  10. 2026Validity-Aware Jailbreak Evaluation for Large Language Modelscited by 0
  11. 2026LLJ Cards: Best practices for the Use of LLMs as Judgescited by 0