mlsanityData to 7 Oct 2026

Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations

arXiv 2410.09097 ↗First posted 9 Oct 2024cs.CL cs.AIcited by 8

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks, but their vulnerability to jailbreak attacks poses significant security risks. This survey paper presents a comprehensive analysis of recent advancements in attack strategies and defense mechanisms within the field of Large Language Model (LLM) red-teaming. We analyze various attack methods, including gradient-based optimization, reinforcement learning, and prompt engineering approaches. We discuss the implications of these attacks on LLM safety and the need for improved defense mechanisms. This work aims to provide a thorough understanding of the current landscape of red-teaming attacks and defenses on LLMs, enabling the development of more secure and reliable language models.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 78 of its 118 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

78 of 118 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique20 Aug
  2. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent23 Jul
  3. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts21 Jul
  4. 2024Get my drift? Catching LLM Task Drift with Activation Deltas2 Jun
  5. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuning28 May
  6. 2024Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teaming21 May
  7. 2024Learn to Disguise: Avoid Refusal Responses in LLM's Defense via a Multi-agent Attacker-Disguiser Game3 Apr
  8. 2024Defending Against Indirect Prompt Injection Attacks With Spotlighting20 Mar
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  10. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  11. 2023AI Control: Improving Safety Despite Intentional Subversion12 Dec
  12. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition24 Oct
8 more
  1. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Models19 Oct
  2. 2023No Offense Taken: Eliciting Offensiveness from Language Models2 Oct
  3. 2023Goal-Oriented Prompt Attack and Safety Evaluation for LLMs21 Sep
  4. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
  5. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection17 Aug
  6. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  7. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimization27 May
  8. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 58

  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  4. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  7. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  8. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  9. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  10. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  11. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  12. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
38 more
  1. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
  2. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 103
  3. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  4. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 91
  5. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 87
  6. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  7. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  8. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  9. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 62
  10. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  11. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 54
  12. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 49
  13. 2023FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Modelscited by 46
  14. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Modelscited by 44
  15. 2024Distract Large Language Models for Automatic Jailbreak Attackcited by 40
  16. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  17. 2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Informationcited by 32
  18. 2024ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Userscited by 32
  19. 2024ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddingscited by 31
  20. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Responsecited by 28
  21. 2024GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanationcited by 24
  22. 2024Prompt Leakage effect and defense strategies for multi-turn LLM interactionscited by 23
  23. 2024h4rm3l: A language for Composable Jailbreak Attack Synthesiscited by 23
  24. 2024Assessing Adversarial Robustness of Large Language Models: An Empirical Studycited by 21
  25. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMscited by 17
  26. 2023GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Modelscited by 16
  27. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 16
  28. 2024Virtual Context: Enhancing Jailbreak Attacks with Special Token Injectioncited by 16
  29. 2022PromptAttack: Prompt-based Attack for Language Models via Gradient Searchcited by 15
  30. 2022A Prompting-based Approach for Adversarial Example Generation and Robustness Enhancementcited by 12
  31. 2024Rapid Optimization for Jailbreaking LLMs via Subconscious Exploitation and Echopraxiacited by 12
  32. 2024AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lenscited by 12
  33. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settingscited by 8
  34. 2024Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Modelscited by 8
  35. 2024AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMscited by 8
  36. 2024StructuralSleight: Automated Jailbreak Attacks on Large Language Models Utilizing Uncommon Text-Organization Structurescited by 7
  37. 2024Reasoning as a Weapon: Adaptive Dual-Path Jailbreak Attack on Large Language Modelscited by 7
  38. 2023Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Modelscited by 6

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  2. 2025SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challengescited by 1
  3. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  4. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  5. 2026TrailBlazer: History-Guided Reinforcement Learning for Black-Box LLM Jailbreakingcited by 1
  6. 2025Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreakingcited by 0
  7. 2025Replicating TEMPEST at Scale: Multi-Turn Adversarial Attacks Against Trillion-Parameter Frontier Modelscited by 0
  8. 2026LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systemscited by 0