mlsanityData to 7 Oct 2026

Attack Prompt Generation for Red Teaming and Defending Large Language Models

arXiv 2310.12505 ↗First posted 19 Oct 2023cs.CL cs.CR cs.LGcited by 73

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. Previous research constructs attack prompts via manual or automatic methods, which have their own limitations on construction cost and quality. To address these issues, we propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts. Specifically, considering the impressive capabilities of newly emerged LLMs, we propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning. Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework to enhance their safety against red teaming attacks. Extensive experiments on different LLMs validate the effectiveness of our proposed attack and defense frameworks. Additionally, we release a series of attack prompts datasets named SAP with varying sizes, facilitating the safety evaluation and enhancement of more LLMs. Our code and dataset is available on https://github.com/Aatrox103/SAP .

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 26 of its 40 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

26 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPT11 Apr
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  4. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
10 more
  1. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  4. 2022A Survey on In-context Learningcited by 717
  5. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  6. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  7. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  8. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  9. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 38
  10. 2022Leashing the Inner Demons: Self-Detoxification for Language Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 33; 2025: 30; 2026: 8.

73 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  3. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  4. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  5. 2024Safeguarding Large Language Models: A Surveycited by 73
  6. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  7. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  8. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  9. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 63
  10. 2024Building Guardrails for Large Language Modelscited by 61
  11. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  12. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
38 more
  1. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  2. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  3. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attackscited by 28
  4. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 26
  5. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teamingcited by 20
  6. 2024LLM-Generated Black-box Explanations Can Be Adversarially Helpfulcited by 20
  7. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  8. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  9. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  10. 2024MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usabilitycited by 14
  11. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  12. 2024You Know What I'm Saying: Jailbreak Attack via Implicit Referencecited by 13
  13. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  14. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  15. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  16. 2024PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routingcited by 10
  17. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  18. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  19. 2025Activated LoRA: Fine-tuned LLMs for Intrinsicscited by 9
  20. 2024Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intentcited by 8
  21. 2024In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Modelscited by 8
  22. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  23. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 8
  24. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  25. 2024SOS! Soft Prompt Attack Against Open-Source Large Language Modelscited by 7
  26. 2025Gandalf the Red: Adaptive Security for LLMscited by 7
  27. 2025Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMscited by 7
  28. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  29. 2025Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraintcited by 6
  30. 2026HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defensecited by 6
  31. 2026Knowledge-Driven Multi-Turn Jailbreaking on Large Language Modelscited by 6
  32. 2024Learn to Disguise: Avoid Refusal Responses in LLM's Defense via a Multi-agent Attacker-Disguiser Gamecited by 5
  33. 2024FairMonitor: A Dual-framework for Detecting Stereotypes and Biases in Large Language Modelscited by 5
  34. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  35. 2025Watermarking Degrades Alignment in Language Models: Analysis and Mitigationcited by 3
  36. 2025Enterprise AI Must Enforce Participant-Aware Access Controlcited by 3
  37. 2025RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contextscited by 3
  38. 2026RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Modelscited by 3

50 most cited of 73 All 73 (JSON)