mlsanityData to 14 Oct 2026

SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

arXiv 2410.07471 ↗First posted 9 Oct 2024cs.LG cs.AI cs.CLcited by 64

Not on the map. Why?

Cited by 26 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs). However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5% and 9.7% win rate increase compared to random selection respectively on Llama-3-8b-Instruct and Merlinite-7b models. Our code is available on github https://github.com/hanshen95/SEAL.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 74 of its 88 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

74 of 88 references.

References · 74

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
38 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  13. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  14. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  15. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  16. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  17. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  18. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  19. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  20. 2018Approximation Methods for Bilevel Programmingcited by 236
  21. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  22. 2016Hyperparameter optimization with approximate gradientcited by 233
  23. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  24. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  25. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  26. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  27. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
  28. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  29. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  30. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  31. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  32. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  33. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  34. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  35. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  36. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  37. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  38. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 28.

64 papers.

Sort:

By area:

  1. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 30
  2. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  3. 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
  4. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  5. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  6. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 20
  7. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 19
  8. 2025Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Modelscited by 17
  9. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  10. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  11. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  12. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
38 more
  1. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  2. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  3. 2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
  4. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
  5. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  6. 2025LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Controlcited by 7
  7. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 6
  8. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  9. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  10. 2025LoRA is All You Need for Safety Alignment of Reasoning LLMscited by 6
  11. 2025Read the Scene, Not the Script: Outcome-Aware Safety for LLMscited by 6
  12. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuningcited by 6
  13. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  14. 2025A First-order Generative Bilevel Optimization Framework for Diffusion Modelscited by 4
  15. 2025A Survey of LLM × DATAcited by 4
  16. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  17. 2025BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretrainingcited by 3
  18. 2026Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safetycited by 3
  19. 2026Token-level Data Selection for Safe LLM Fine-tuningcited by 3
  20. 2025Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shiftcited by 2
  21. 2025Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generationcited by 2
  22. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignmentcited by 2
  23. 2026NeST: Neuron Selective Tuning for LLM Safetycited by 2
  24. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuningcited by 1
  25. 2025Faster Gradient Methods for Highly-Smooth Stochastic Bilevel Optimizationcited by 1
  26. 2025Nonconvex Decentralized Stochastic Bilevel Optimization under Heavy-Tailed Noisecited by 1
  27. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  28. 2025Preventing Catastrophic Forgetting: Behavior-Aware Sampling for Safer Language Model Fine-Tuningcited by 1
  29. 2025Alchemist: Unlocking Efficiency in Text-to-Image Model Training via Meta-Gradient Data Selectioncited by 1
  30. 2026DeFrame: Debiasing Large Language Models Against Framing Effectscited by 1
  31. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
  32. 2026A Tale of Two Problems: Multi-Task Bilevel Learning Meets Equality Constrained Multi-Objective Optimizationcited by 1
  33. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directionscited by 1
  34. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimizationcited by 1
  35. 2026DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuningcited by 1
  36. 2025Bayesian Optimization for Enhanced Language Models: Optimizing Acquisition Functionscited by 0
  37. 2025EnchTable: Unified Safety Alignment Transfer in Fine-tuned Large Language Modelscited by 0
  38. 2026TBDFiltering: Sample-Efficient Tree-Based Data Filteringcited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing LLM Safe Alignment with Safety Representation Ranking
  2. 2025Curvature-Aware Safety Restoration In LLMs Fine-Tuning
  3. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  4. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  5. 2024Locking Down the Finetuned LLMs Safety
  6. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
  7. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  8. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training