Towards Understanding the Influence of Reward Margin on Preference Model Performance
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Reinforcement Learning from Human Feedback (RLHF) is a widely used framework for the training of language models. However, the process of using RLHF to develop a language model that is well-aligned presents challenges, especially when it comes to optimizing the reward model. Our research has found that existing reward models, when trained using the traditional ranking objective based on human preference data, often struggle to effectively distinguish between responses that are more or less favorable in real-world scenarios. To bridge this gap, our study introduces a novel method to estimate the preference differences without the need for detailed, exhaustive labels from human annotators. Our experimental results provide empirical evidence that incorporating margin values into the training process significantly improves the effectiveness of reward models. This comparative analysis not only demonstrates the superiority of our approach in terms of reward prediction accuracy but also highlights its effectiveness in practical applications.
Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 25 references that are in the map. First posted on arXiv 7 Apr 2024.
Builds on
21 of 25 references.
Closest ancestors · 12
Same direction, newest first
- 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
- 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
- 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
- 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
- 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
- 2022Scaling Laws for Reward Model Overoptimization19 Oct
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
- 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
- 2019Fine-Tuning Language Models from Human Preferences18 Sep
- 2017Deep reinforcement learning from human preferences12 Jun
Other references · 9
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Learning to summarize from human feedbackcited by 2,588
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
- 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
- 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
- 2021Consequences of Misaligned AIcited by 80
Built on by
Citing papers per year: 2024: 2; 2025: 6; 2026: 1.
9 papers.
- 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
- 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Modelscited by 19
- 2024Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasetscited by 5
- 2025Controllable Protein Sequence Generation with LLM Preference Optimizationcited by 5
- 2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguitycited by 2
- 2025AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignmentcited by 2
- 2025Adaptive Margin RLHF via Preference over Preferencescited by 1
- 2025UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hackingcited by 0
- 2026Personalized Alignment Revisited: The Necessity and Sufficiency of User Diversitycited by 0
- 2026Personalized Alignment Revisited: The Necessity and Sufficiency of User Diversity9 May 2026
- 2025UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking30 Nov 2025
- 2025AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment12 Nov 2025
- 2025Adaptive Margin RLHF via Preference over Preferences26 Sep 2025
- 2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity29 May 2025
- 2025What Makes a Reward Model a Good Teacher? An Optimization Perspective19 Mar 2025
- 2025Controllable Protein Sequence Generation with LLM Preference Optimization25 Jan 2025
- 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models9 Oct 2024
- 2024Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets12 Jun 2024
Similar, not cited
Close in wording, no citation either way
- 2024How to Evaluate Reward Models for RLHF
- 2024Towards Comprehensive Preference Data Collection for Reward Modeling
- 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
- 2025A Systematic Analysis of Base Model Choice for Reward Modeling
- 2025Influencing Humans to Conform to Preference Models for RLHF
- 2023The History and Risks of Reinforcement Learning and Human Feedback
- 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
- 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning