mlsanityData to 7 Oct 2026

Spectral Souping: A Unified Framework for Online Preference Alignment

arXiv 2605.20408 ↗First posted 19 May 2026cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) effectively aligns Large Language Models (LLMs) with aggregate human preferences but often fails to address the diverse and conflicting needs of individual users. To overcome this issue, we introduce Spectral Souping, a unified framework for efficient, online preference alignment. Our contribution is the discovery of a universal spectral representation within LLMs, which is proven to be highly amenable to model merging. This theoretical insight enables a two-phase methodology: we first learn a basis of specialized policies offline, each focused on a distinct, fine-grained preference dimension. An online adaptation algorithm then efficiently ``soups'' these policies at inference time, either by merging their outputs or parameters, enabling rapid model adaptation without the need for costly online retraining w.r.t. tailored preference rewards. Experiments on online preference alignment benchmarks demonstrate that our method achieves significant performance improvements over existing state-of-the-art approaches, presenting a scalable and computationally efficient solution for dynamically adapting LLMs to individual user preferences.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 63 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

54 of 63 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment7 Jan
  2. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  4. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  5. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  6. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  7. 2023On Diversified Preferences of Large Language Model Alignment12 Dec
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2019Introduction to Online Convex Optimizationcited by 793
  10. 2025Process Reinforcement through Implicit Rewardscited by 478
  11. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  12. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
34 more
  1. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  2. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  3. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  4. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  5. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  6. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  7. 2023Controlled Decoding from Language Modelscited by 133
  8. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  9. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  10. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  11. 2024Knowledge Fusion of Large Language Modelscited by 110
  12. 2023Aligning Language Models to User Opinionscited by 98
  13. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  14. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  15. 2019Bayesian Symbolic Regressioncited by 77
  16. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  17. 2016Deep Successor Reinforcement Learningcited by 72
  18. 2023LoRA ensembles for large language model fine-tuningcited by 65
  19. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  20. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  21. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  22. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  23. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  24. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  25. 2024Dissecting Human and LLM Preferencescited by 17
  26. 2024MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimizationcited by 16
  27. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  28. 2023Provable Representation with Efficient Planning for Partial Observable Reinforcement Learningcited by 10
  29. 2024Preference Adaptive and Sequential Text-to-Image Generationcited by 8
  30. 2025From Prompting to Alignment: A Generative Framework for Query Recommendationcited by 7
  31. 2024Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answeringcited by 4
  32. 2025Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactionscited by 4
  33. 2025Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learningcited by 3
  34. 2025ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Makingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Online Preference Alignment for Language Models via Count-based Exploration
  2. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  3. 2024OPTune: Efficient Online Preference Tuning
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  5. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  6. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  7. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
  8. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?