mlsanityData to 14 Oct 2026

Lifelong Safety Alignment for Language Models

arXiv 2505.20259 ↗First posted 26 May 2025cs.CR cs.AI cs.CLcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

LLMs have made impressive progress, but their growing capabilities also expose them to highly flexible jailbreaking attacks designed to bypass safety alignment. While many existing defenses focus on known types of attacks, it is more critical to prepare LLMs for unseen attacks that may arise during deployment. To address this, we propose a lifelong safety alignment framework that enables LLMs to continuously adapt to new and evolving jailbreaking strategies. Our framework introduces a competitive setup between two components: a Meta-Attacker, trained to actively discover novel jailbreaking strategies, and a Defender, trained to resist them. To effectively warm up the Meta-Attacker, we first leverage the GPT-4o API to extract key insights from a large collection of jailbreak-related research papers. Through iterative training, the first iteration Meta-Attacker achieves a 73% attack success rate (ASR) on RR and a 57% transfer ASR on LAT using only single-turn attacks. Meanwhile, the Defender progressively improves its robustness and ultimately reduces the Meta-Attacker's success rate to just 7%, enabling safer and more reliable deployment of LLMs in open-ended environments. The code is available at https://github.com/sail-sg/LifelongSafetyAlignment.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 80 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

64 of 80 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025STAIR: Improving Safety Alignment with Introspective Reasoning4 Feb
  2. 2024Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment5 Nov
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  4. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  5. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 56

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2024Qwen2 Technical Reportcited by 2,120
38 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  11. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  13. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  14. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  15. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  16. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  17. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  18. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  19. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  20. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  21. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  22. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  23. 2020Bag of Tricks for Adversarial Trainingcited by 200
  24. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  25. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  26. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 136
  27. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  28. 2018Adversarial Attacks and Defences Competitioncited by 96
  29. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  30. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79
  31. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 76
  32. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  33. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  34. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 56
  35. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 42
  36. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systemscited by 39
  37. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
  38. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  2. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generationcited by 14
  3. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimizationcited by 14
  4. 2025Safety Alignment of LMs via Non-cooperative Gamescited by 8
  5. 2025Adversarial D'ej`a Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attackscited by 7
  6. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6
  7. 2026DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protectioncited by 6
  8. 2025Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimizationcited by 4
  9. 2026COPA: Continual Preference Optimization for Adaptive Prompt Injection Defensecited by 2
  10. 2026CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinementcited by 2
  11. 2025Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Modelscited by 0
  12. 2026Characterizing Model-Native Skillscited by 0
3 more
  1. 2026A Systematic Investigation of RL-Jailbreaking in LLMscited by 0
  2. 2026REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreakcited by 0
  3. 2026HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  2. 2025Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges
  3. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs
  4. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  5. 2025AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender
  6. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  7. 2026SecureBreak -- A dataset towards safe and secure models
  8. 2026Jailbreaking LLMs via Calibration