mlsanityData to 7 Oct 2026

Soft Preference Optimization: Aligning Language Models to Expert Distributions

arXiv 2405.00747 ↗First posted 30 Apr 2024cs.LG cs.AIcited by 9

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 38 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

27 of 38 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  2. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  3. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  4. 2024Direct Preference Optimization with an Offset16 Feb
  5. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts12 Feb
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  7. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  8. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  9. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  11. 2023Nash Learning from Human Feedbackcited by 220
  12. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
4 more
  1. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  2. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  3. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  4. 2016Unsupervised Pretraining for Sequence to Sequence Learningcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexitycited by 11
  3. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
  4. 2025Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixingcited by 2
  5. 2026Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videoscited by 1
  6. 2025Fusing Rewards and Preferences in Reinforcement Learningcited by 0
  7. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoningcited by 0
  8. 2026Uncertainty-Normalized Margins for Direct Preference Optimizationcited by 0
  9. 2026Robust Nash Alignment under Preference Uncertaintycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  5. 2025Offline Preference Optimization via Maximum Marginal Likelihood Estimation
  6. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  7. 2025Aligning LLMs with Domain Invariant Reward Models
  8. 2024Transforming and Combining Rewards for Aligning Large Language Models