mlsanityData to 14 Oct 2026

Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment

arXiv 2601.07200 ↗First posted 12 Jan 2026cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The inherent safety alignment of Large Language Models (LLMs) is prone to erosion during fine-tuning, even when using seemingly innocuous datasets. While existing defenses attempt to mitigate this via data selection, they typically rely on heuristic, instance-level assessments that neglect the global geometry of the data distribution and fail to explicitly repel harmful patterns. To address this, we introduce Safety Optimal Transport (SOT), a novel framework that reframes safe fine-tuning from an instance-level filtering challenge to a distribution-level alignment task grounded in Optimal Transport (OT). At its core is a dual-reference ``push-pull'' weight-learning mechanism: SOT optimizes sample importance by actively pulling the downstream distribution towards a trusted safe anchor while simultaneously pushing it away from a general harmful reference. This establishes a robust geometric safety boundary that effectively purifies the training data. Extensive experiments across diverse model families and domains demonstrate that SOT significantly enhances model safety while maintaining competitive downstream performance, achieving a superior safety-utility trade-off compared to baselines.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 61 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

41 of 61 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025SDD: Self-Degraded Defense against Malicious Fine-tuning27 Jul
  2. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment24 Jul
  3. 2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning4 Jun
  4. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging21 Mar
  5. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security30 Aug
  6. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  7. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  8. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  9. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  11. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  10. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  11. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  12. 2023Data Selection for Language Models via Importance Resamplingcited by 329
17 more
  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  2. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  3. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  4. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  5. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  6. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  7. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  8. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 21
  9. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  10. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 18
  11. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  12. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
  13. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  14. 2025Alleviating the Fear of Losing Alignment in LLM Fine-tuningcited by 9
  15. 2025Model Immunization from a Condition Number Perspectivecited by 8
  16. 2024Atoxia: Red-teaming Large Language Models with Target Toxic Answerscited by 7
  17. 2025Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigmcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibrationcited by 1
  2. 2026VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  2. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  3. 2024Locking Down the Finetuned LLMs Safety
  4. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  5. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  6. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  7. 2026Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks
  8. 2025Secure LLM Fine-Tuning via Safety-Aware Probing