mlsanityData to 14 Oct 2026

Geometric-Averaged Preference Optimization for Soft Preference Labels

arXiv 2409.06691 ↗First posted 10 Sep 2024cs.LG cs.AI cs.CLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Many algorithms for aligning LLMs with human preferences assume that human preferences are binary and deterministic. However, human preferences can vary across individuals, and therefore should be represented distributionally. In this work, we introduce the distributional soft preference labels and improve Direct Preference Optimization (DPO) with a weighted geometric average of the LLM output likelihood in the loss function. This approach adjusts the scale of learning loss based on the soft labels such that the loss would approach zero when the responses are closer to equally preferred. This simple modification can be easily applied to any DPO-based methods and mitigate over-optimization and objective mismatch, which prior works suffer from. Our experiments simulate the soft preference labels with AI feedback from LLMs and demonstrate that geometric averaging consistently improves performance on standard benchmarks for alignment research. In particular, we observe more preferable responses than binary labels and significant improvements where modestly-confident labels are in the majority.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 66 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

56 of 66 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  2. 2024Filtered Direct Preference Optimization22 Apr
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  5. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  6. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  7. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  8. 2024Aligning Crowd Feedback via Distributional Preference Reward Modeling15 Feb
  9. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  10. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023Aligning Language Models with Human Preferences via a Bayesian Approach9 Oct
4 more
  1. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2023Preference Ranking Optimization for Human Alignment30 Jun
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,085
28 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  5. 2023Instruction Tuning with GPT-4cited by 686
  6. 2024Self-Rewarding Language Modelscited by 681
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  8. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  11. 2020Critic Regularized Regressioncited by 275
  12. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  13. 2024Iterative Reasoning Preference Optimizationcited by 246
  14. 2023Nash Learning from Human Feedbackcited by 224
  15. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  16. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  17. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  18. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  19. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  20. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 146
  21. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  22. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  23. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  24. 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
  25. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  26. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 64
  27. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 28
  28. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 5.

18 papers.

Sort:

By area:

  1. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  2. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  3. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  4. 2025A Survey of Direct Preference Optimizationcited by 36
  5. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 27
  6. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
  7. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 14
  8. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  9. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  10. 2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
  11. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimizationcited by 3
  12. 2026Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videoscited by 3
6 more
  1. 2026Gaze patterns predict preference and confidence in pairwise AI image evaluationcited by 1
  2. 2025References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulationcited by 0
  3. 2025Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processescited by 0
  4. 2026Disentangling Optimization Scale from Preference Scale in DPOcited by 0
  5. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillationcited by 0
  6. 2026Robust Nash Alignment under Preference Uncertaintycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
  2. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment
  3. 2026Response Time Enhances Alignment with Heterogeneous Preferences
  4. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  5. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  6. 2025HPS: Hard Preference Sampling for Human Preference Alignment
  7. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
  8. 2025Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback