Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolution
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
Large language models remain vulnerable to adversarial prompts that elicit harmful outputs. Existing safety paradigms typically couple red-teaming and post-training in a closed, policy-centric loop, causing attack discovery to suffer from rapid saturation and limiting the exposure of novel failure modes, while leaving defenses inefficient, rigid, and difficult to transfer across victim models. To this end, we propose EvoSafety, an LLM safety framework built around persistent, inspectable, and reusable external structures. For red teaming, EvoSafety equips the attack policy with an adversarial skill library, enabling continued vulnerability probing through simple library expansion after saturation, while supporting the evolution of adversarial vectors. For defense learning, EvoSafety replaces model-specific safety fine-tuning with a lightweight auxiliary defense model augmented with memory retrieval. This enables efficient, transferable, and model-agnostic safety improvements, while allowing robustness to be enhanced solely through memory updates. With a single training procedure, the defense policy can operate in both Steer and Guard modes: the former activates the victim model's intrinsic defense mechanisms, while the latter directly filters harmful inputs. Extensive experiments demonstrate the superiority of EvoSafety: in Guard mode, it achieves a 99.61% defense success rate, outperforming Qwen3Guard-8B by 14.13% with only 37.5% of its parameters, while preserving reasoning performance on benign queries. Warning: This paper contains potentially harmful text.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 88 of its 103 references that are in the map. First posted on arXiv 13 May 2026.
Builds on
88 of 103 references.
Closest ancestors · 14
Same direction, newest first
- 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety2 Feb
- 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay15 Jan
- 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection6 Oct
- 2025RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents2 Oct
- 2025Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning1 Jun
- 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration20 Mar
- 2025OpenAI's Approach to External Red Teaming for AI Models and Systems24 Jan
- 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent23 Jul
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
- 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
- 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
- 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked14 Aug
2 more
Other references · 74
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
- 2023Qwen Technical Reportcited by 3,273
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
- 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
- 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
- 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
- 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
- 2023Cognitive Architectures for Language Agentscited by 422
- 2023Detecting Language Model Attacks with Perplexitycited by 411
- 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
- 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
- 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
- 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
- 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
- 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
- 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
- 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
- 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
- 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
- 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
- 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
- 2025Qwen3Guard Technical Reportcited by 125
- 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
- 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
- 2025A Survey on Code Generation with LLM-based Agentscited by 104
- 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
- 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
50 most cited of 74 All 74 (JSON)
Built on by
Not cited yet.