mlsanityData to 14 Oct 2026

Preference Tuning as Spectral Update Reorganization

arXiv 2607.20438 ↗First posted 12 May 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Preference-based post-training is usually understood through endpoint behavior, yet the learned update that produces this behavior remains largely opaque. We study RLHF and related preference optimization through the spectral structure of their induced parameter updates. By decomposing effective LoRA updates and reloading their spectral components as plug-in modules, we turn preference-induced updates into objects that can be isolated, recomposed, and directly intervened on. Across model families, optimization algorithms, and supervision regimes, these updates consistently develop a spectral head--tail organization. A compact head emerges early and carries the dominant endpoint shift, while a heterogeneous residual tail remains. The split is functional rather than merely descriptive. Plug-in intervention shows that the head accounts for the visible behavioral departure from the base model, while the tail is weak in isolation. Cross-run recomposition further shows that mixed adapters follow the source of the head, indicating that the head carries run-level solver bias. This endpoint dominance does not imply learning sufficiency. Head-only learning is non-vacuous but fails to recover the full solution, especially on out-of-distribution behavior. Tail-only learning yields little visible gain, yet the full solution is not recovered without the tail. These findings recast preference post-training as structured update reorganization rather than a monolithic behavioral correction, and suggest that alignment gain and coverage loss are tied to how the learned update itself is organized.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

28 of 35 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  9. 2022Editing Models with Task Arithmeticcited by 1,120
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
16 more
  1. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  2. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  3. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  4. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  5. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 51
  6. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  7. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  8. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  9. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 29
  10. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  11. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 13
  12. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selectioncited by 12
  13. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learningcited by 10
  14. 2025Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledgecited by 6
  15. 2026The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVRcited by 3
  16. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Spectral Souping: A Unified Framework for Online Preference Alignment
  2. 2026Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
  3. 2026Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs
  4. 2026Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
  5. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
  6. 2026LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment
  7. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  8. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences