mlsanityData to 7 Oct 2026

Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolution

arXiv 2605.13411 ↗First posted 13 May 2026cs.CR cs.CLcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large language models remain vulnerable to adversarial prompts that elicit harmful outputs. Existing safety paradigms typically couple red-teaming and post-training in a closed, policy-centric loop, causing attack discovery to suffer from rapid saturation and limiting the exposure of novel failure modes, while leaving defenses inefficient, rigid, and difficult to transfer across victim models. To this end, we propose EvoSafety, an LLM safety framework built around persistent, inspectable, and reusable external structures. For red teaming, EvoSafety equips the attack policy with an adversarial skill library, enabling continued vulnerability probing through simple library expansion after saturation, while supporting the evolution of adversarial vectors. For defense learning, EvoSafety replaces model-specific safety fine-tuning with a lightweight auxiliary defense model augmented with memory retrieval. This enables efficient, transferable, and model-agnostic safety improvements, while allowing robustness to be enhanced solely through memory updates. With a single training procedure, the defense policy can operate in both Steer and Guard modes: the former activates the victim model's intrinsic defense mechanisms, while the latter directly filters harmful inputs. Extensive experiments demonstrate the superiority of EvoSafety: in Guard mode, it achieves a 99.61% defense success rate, outperforming Qwen3Guard-8B by 14.13% with only 37.5% of its parameters, while preserving reasoning performance on benign queries. Warning: This paper contains potentially harmful text.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 88 of its 103 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

88 of 103 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety2 Feb
  2. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay15 Jan
  3. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection6 Oct
  4. 2025RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents2 Oct
  5. 2025Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning1 Jun
  6. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration20 Mar
  7. 2025OpenAI's Approach to External Red Teaming for AI Models and Systems24 Jan
  8. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent23 Jul
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  10. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  11. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
  12. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked14 Aug
2 more
  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  2. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  13. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  14. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  15. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  16. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  17. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  18. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  19. 2023Cognitive Architectures for Language Agentscited by 422
  20. 2023Detecting Language Model Attacks with Perplexitycited by 411
  21. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  22. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  23. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  24. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  25. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  26. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  27. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  28. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  29. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  30. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  31. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  32. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  33. 2025Qwen3Guard Technical Reportcited by 125
  34. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  35. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  36. 2025A Survey on Code Generation with LLM-based Agentscited by 104
  37. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  38. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88

50 most cited of 74 All 74 (JSON)

Built on by

Not cited yet.