mlsanityData to 7 Oct 2026

GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning

arXiv 2603.10243 ↗First posted 10 Mar 2026cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent studies show that the safety alignment of large language models (LLMs) can be easily compromised even by seemingly non-adversarial fine-tuning. To preserve safety alignment during fine-tuning, a widely used strategy is to jointly optimize safety and task objectives by mixing in the original alignment data, which is typically inaccessible even for open-weight LLMs. Inspired by generative replay in continual learning, we propose Generative Replay for Safety Alignment Preservation (GR-SAP), a unified framework that synthesizes domain-specific alignment data from LLMs and integrate them during downstream adaption to preserve safety alignment. Theoretical and empirical analyses demonstrate this synthetic data serves as a reliable proxy for the original alignment data. Experiments across various models and downstream tasks show that GR-SAP substantially mitigates fine-tuning-induced safety degradation while maintaining comparable downstream performance. Our code is available at https://github.com/chili-lab/gr-sap.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 104 of its 121 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

104 of 121 references.

Closest ancestors · 25

Same direction, newest first

  1. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning10 Dec
  2. 2025Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs3 Oct
  3. 2025SDD: Self-Degraded Defense against Malicious Fine-tuning27 Jul
  4. 2025Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs21 Jun
  5. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging21 Mar
  6. 2025Improving LLM Safety Alignment with Dual-Objective Optimization5 Mar
  7. 2025Safe at the Margins: A General Approach to Safety Alignment in Low-Resource English Languages -- A Singlish Case Study18 Feb
  8. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
  9. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements11 Oct
  10. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security30 Aug
  11. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  12. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
13 more
  1. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models12 Jun
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curation24 May
  4. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  5. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  6. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  9. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  10. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  13. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 79

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2017Continual Learning with Deep Generative Replaycited by 1,152
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  13. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  14. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  15. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  16. 2023Textbooks Are All You Needcited by 561
  17. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  18. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  19. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  20. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  21. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  22. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  23. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  24. 2024The Faiss librarycited by 370
  25. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  26. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  27. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  28. 20242 OLMo 2 Furiouscited by 302
  29. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  30. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  31. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  32. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  33. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  34. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  35. 2023Model evaluation for extreme riskscited by 167
  36. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  37. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  38. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Trustworthiness Costs of Domain Adaptation in Small Language Models:A Cross-Architecture Empirical Studycited by 0
  2. 2026DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  2. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  3. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
  4. 2026SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models
  5. 2026Continual Safety Alignment via Gradient-Based Sample Selection
  6. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  7. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  8. 2025Backtracking for Safety