mlsanityData to 14 Oct 2026

Two to Tango: Coupled Task-Reference Selection for Safe LLM Fine-tuning

arXiv 2606.09866 ↗First posted 1 Jun 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Fine-tuning safety aligned large language models (LLMs) on downstream data improves adaptation but may erode learned safety behavior. Existing methods use fixed safety examples, global constraints, or one-sided task filtering. Our diagnostics show task updates expose different safety constraints, motivating joint selection of relevant references and compatible task samples. We propose DualSelect, a coupled framework for task and reference selection that refreshes task conditioned safety references before filtering whole task samples compatible with the induced reference direction. Under a minimax view, DualSelect selects safety references with high preservation loss and task conflict, together with compatible task samples, through entropy-regularized scoring surrogates, lazy reference refresh, and gradient correction. On 1B-8B LLMs, DualSelect preserves safety without losing task utility; using the REDORCA judge, it improves Safety Avg. over the strongest baseline by at least 5.10 points and remains highest in Safety Avg. across judges with moderate overhead. This view extends to retention focused continual learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 50 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

21 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion19 Jan
  2. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models15 Dec
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  8. 2018Experience Replay for Continual Learningcited by 1,095
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  10. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  12. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
6 more
  1. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  2. 2016Hyperparameter optimization with approximate gradientcited by 233
  3. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 15
  4. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  5. 2026GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometrycited by 8
  6. 2025Learn More, Forget Less: A Gradient-Aware Data Selection Approach for LLMcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning
  2. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  3. 2026Continual Safety Alignment via Gradient-Based Sample Selection
  4. 2026Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models
  5. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  6. 2026Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraints
  7. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning
  8. 2025Secure LLM Fine-Tuning via Safety-Aware Probing