mlsanityData to 7 Oct 2026

Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)

arXiv 2407.14937 ↗First posted 20 Jul 2024cs.CL cs.CRcited by 44

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Creating secure and resilient applications with large language models (LLM) requires anticipating, adjusting to, and countering unforeseen threats. Red-teaming has emerged as a critical technique for identifying vulnerabilities in real-world LLM implementations. This paper presents a detailed threat model and provides a systematization of knowledge (SoK) of red-teaming attacks on LLMs. We develop a taxonomy of attacks based on the stages of the LLM development and deployment process and extract various insights from previous research. In addition, we compile methods for defense and practical red-teaming strategies for practitioners. By delineating prominent attack motifs and shedding light on various entry points, this paper provides a framework for improving the security and robustness of LLM-based systems.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 244 of its 376 references that are in the map. First posted on arXiv 20 Jul 2024.

Builds on

244 of 376 references.

Closest ancestors · 30

Same direction, newest first

  1. 2024Automated Progressive Red Teaming4 Jul
  2. 2024Purple-teaming LLMs with Adversarial Defender Training1 Jul
  3. 2024Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teaming21 May
  4. 2024Benchmark Early and Red Team Often: A Framework for Assessing and Managing Dual-Use Hazards of AI Foundation Models15 May
  5. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  6. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Models31 Mar
  7. 2024Defending Against Indirect Prompt Injection Attacks With Spotlighting20 Mar
  8. 2024Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications5 Mar
  9. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  10. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  11. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents15 Feb
  12. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
18 more
  1. 2024Gradient-Based Language Model Red Teaming30 Jan
  2. 2024Towards Red Teaming in Multimodal and Multilingual Translation29 Jan
  3. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications15 Jan
  4. 2023Hijacking Large Language Models via Adversarial In-Context Learning16 Nov
  5. 2023JAB: Joint Adversarial Prompting and Belief Augmentation16 Nov
  6. 2023AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications14 Nov
  7. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  8. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming10 Nov
  9. 2023Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases22 Oct
  10. 2023Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games30 Sep
  11. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
  12. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked14 Aug
  13. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  14. 2023Seeing Seeds Beyond Weeds: Green Teaming Generative AI for Beneficial Uses30 May
  15. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimization27 May
  16. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  17. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity30 Jan
  18. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 214

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
38 more
  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  13. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  14. 2021Ethical and social risks of harm from Language Modelscited by 957
  15. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  16. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  17. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  18. 2023Steering Language Models With Activation Engineeringcited by 824
  19. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  20. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  21. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  22. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  23. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  24. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  25. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  26. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  27. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  28. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  29. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  30. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  31. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  32. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  33. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  34. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
  35. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  36. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  37. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  38. 2020Backdoor Learning: A Surveycited by 412

50 most cited of 214 All 214 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 29; 2026: 8.

44 papers.

Sort:

By area:

  1. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attackscited by 118
  2. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  3. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  4. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  5. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  6. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  7. 2025MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Serverscited by 28
  8. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  9. 2024Permissive Information-Flow Analysis for Large Language Modelscited by 22
  10. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  11. 2024Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You Incited by 21
  12. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
32 more
  1. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  2. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomycited by 14
  3. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  4. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  5. 2025Better Language Model Inversion by Compactly Representing Next-Token Distributionscited by 9
  6. 2025When Autonomy Goes Rogue: Preparing for Risks of Multi-Agent Collusion in Social Systemscited by 9
  7. 2025Gandalf the Red: Adaptive Security for LLMscited by 7
  8. 2025TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agentcited by 7
  9. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 7
  10. 2024SoK: Prompt Hacking of Large Language Modelscited by 5
  11. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 5
  12. 2025Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguardscited by 5
  13. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modelingcited by 5
  14. 2025Understanding and Mitigating Risks of Generative AI in Financial Servicescited by 4
  15. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  16. 2025Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulationcited by 3
  17. 2025Watermarking Degrades Alignment in Language Models: Analysis and Mitigationcited by 3
  18. 2025A Call to Action for a Secure-by-Design Generative AI Paradigmcited by 3
  19. 2025Cloud Investigation Automation Framework (CIAF): An AI-Driven Approach to Cloud Forensicscited by 3
  20. 2025A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellnesscited by 3
  21. 2025LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Userscited by 2
  22. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safetycited by 2
  23. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  24. 2026Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Riskscited by 1
  25. 2026Automating Cloud Security and Forensics Through a Secure-by-Design Generative AI Frameworkcited by 1
  26. 2024Trading Devil Final: Backdoor attack via Stock market and Bayesian Optimizationcited by 0
  27. 2025TombRaider: Entering the Vault of History to Jailbreak Large Language Modelscited by 0
  28. 2025AJF: Adaptive Jailbreak Framework Based on the Comprehension Ability of Black-Box Large Language Modelscited by 0
  29. 2025Quantifying Document Impact in RAG-LLMscited by 0
  30. 2026Black-Box Inference of LLM Architectural Properties with Restrictive API Accesscited by 0
  31. 2026From Adoption to Deployment: A Qualitative Study on AI Integration in Software Development Practicecited by 0
  32. 2026SIR: Self-improving Red-teaming for Compute Use Agentscited by 0