mlsanityData to 14 Oct 2026

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment

arXiv 2410.02197 ↗First posted 3 Oct 2024cs.AI cs.CL cs.LGcited by 27

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Modeling human preferences is crucial for aligning foundation models with human values. Traditional reward modeling methods, such as the Bradley-Terry (BT) reward model, fall short in expressiveness, particularly in addressing intransitive preferences. In this paper, we introduce preference embedding, an approach that embeds responses into a latent space to capture intricate preference structures efficiently, achieving linear query complexity. Additionally, we propose preference score-based General Preference Optimization (GPO), which generalizes reward-based reinforcement learning from human feedback (RLHF). Experimental results show that our General Preference embedding Model (GPM) consistently outperforms the BT reward model on the RewardBench benchmark and effectively models cyclic preferences where any BT reward model behaves like a random guess. Furthermore, evaluations on downstream tasks such as AlpacaEval2.0, following the language model post-training with GPO and our general preference model, reveal performance improvements over BT models. These findings indicate that our method may enhance the alignment of foundation models with nuanced human values. The code is available at https://github.com/general-preference/general-preference-model.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 62 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

38 of 62 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  4. 2024REBEL: Reinforcement Learning via Regressing Relative Rewards25 Apr
  5. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  6. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  7. 2023Nash Learning from Human Feedback1 Dec
  8. 2023Is RLHF More Difficult than Standard RL?25 Jun
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  11. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
14 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  3. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  4. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  5. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  6. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  7. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  8. 2015Contextual Dueling Banditscited by 101
  9. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  10. 2020Real World Games Look Like Spinning Topscited by 66
  11. 2018Re-evaluating Evaluationcited by 47
  12. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23
  13. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  14. 2023Borda Regret Minimization for Generalized Linear Dueling Banditscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 14.

27 papers.

Sort:

By area:

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  3. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 23
  4. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  5. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  6. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  7. 2025Skewed Score: A statistical framework to assess autograderscited by 7
  8. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  9. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  10. 2025ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignmentcited by 3
  11. 2026What Does Preference Learning Recover from Pairwise Comparison Data?cited by 3
  12. 2025Beyond Theorem Proving: Formulation, Framework and Benchmark for Formal Problem-Solvingcited by 2
15 more
  1. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  2. 2025The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curationcited by 2
  3. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 2
  4. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  5. 2026Embeddings for Preferences, Not Semanticscited by 1
  6. 2026General Preference Reinforcement Learningcited by 1
  7. 2026The Representation-Rationalizability Tradeoff in Reward Learningcited by 1
  8. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0
  9. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignmentcited by 0
  10. 2026On Generalizing Preference-based Reinforcement Learning to Multi-Objective Problemscited by 0
  11. 2026MoPLEx: Estimating Plackett-Luce Mixture Models for Multi-Objective Alignmentcited by 0
  12. 2026Learning What to Activate: Combinatorial Capability Allocation for Long-Horizon Multimodal Agentscited by 0
  13. 2026Can You Trust Your Preference Models? Stackelberg Games for Alignmentcited by 0
  14. 2026Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arenacited by 0
  15. 2026When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  2. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  3. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
  4. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
  5. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  6. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2024Transforming and Combining Rewards for Aligning Large Language Models