Data to 7 Oct 2026

Secrets of RLHF in Large Language Models Part II: Reward Modeling

arXiv 2401.06080 ↗First posted 11 Jan 2024cs.AIcited by 170

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that distribution and are not suitable for iterative RLHF training. In this report, we attempt to address these two issues. (1) From a data perspective, we propose a method to measure the strength of preferences within the data, based on a voting mechanism of multiple reward models. Experimental results confirm that data with varying preference strengths have different impacts on reward model performance. We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset and fully leverage high-quality preference data. (2) From an algorithmic standpoint, we introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization. Furthermore, we employ meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples, and this approach can be utilized for iterative RLHF optimization.

Key facts. Cited by 170 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 40 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

33 of 40 references.

Closest ancestors · 11

Same direction, newest first

Other references · 22

2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2019The Curious Case of Neural Text Degenerationcited by 2,881
2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
2020Learning to summarize from human feedbackcited by 2,562
2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
2023A Survey on Evaluation of Large Language Modelscited by 1,227
2023Towards Understanding Sycophancy in Language Modelscited by 988

10 more

2019When Does Label Smoothing Help?cited by 985
2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
2022Improving alignment of dialogue agents via targeted human judgementscited by 527
2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2021Alignment of Language Agentscited by 167
2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
2023Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinementcited by 43
2023Specific versus General Principles for Constitutional AIcited by 41
2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 71; 2025: 66; 2026: 33.

170 papers.

Sort:

By area:

2024Qwen2.5 Technical Reportcited by 4,713
2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
2024Detoxifying Large Language Models via Knowledge Editingcited by 86
2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76

38 more

2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
2024Aligning LLMs with Individual Preferences via Interactioncited by 63
2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
2024Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Explorationcited by 58
2024Learn Your Reference Model for Real Good Alignmentcited by 56
2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 48
2024RaFe: Ranking Feedback Improves Query Rewriting for RAGcited by 47
2025Reward Reasoning Modelcited by 47
2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 30
2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
2025Language Model Personalization via Reward Factorizationcited by 27
2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
2024Weaver: Foundation Models for Creative Writingcited by 25
2024Prototypical Reward Network for Data-Efficient RLHFcited by 22
2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
2024Performance-Aligned LLMs for Generating Fast Codecited by 20
2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 20
2025Improving RL Exploration for LLM Reasoning through Retrospective Replaycited by 19
2024MallowsPO: Fine-Tune Your LLM with Preference Dispersionscited by 18
2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
2024ROPO: Robust Preference Optimization for Large Language Modelscited by 16

50 most cited of 170 All 170 (JSON)

Similar, not cited

Close in wording, no citation either way