mlsanityData to 7 Oct 2026

Fine-Tuning Language Models with Reward Learning on Policy

arXiv 2403.19279 ↗First posted 28 Mar 2024cs.CL cs.AI cs.LGcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as an effective approach to aligning large language models (LLMs) to human preferences. RLHF contains three steps, i.e., human preference collecting, reward learning, and policy optimization, which are usually performed serially. Despite its popularity, however, (fixed) reward models may suffer from inaccurate off-distribution, since policy optimization continuously shifts LLMs' data distribution. Repeatedly collecting new preference data from the latest LLMs may alleviate this issue, which unfortunately makes the resulting system more complicated and difficult to optimize. In this paper, we propose reward learning on policy (RLP), an unsupervised framework that refines a reward model using policy samples to keep it on-distribution. Specifically, an unsupervised multi-view learning method is introduced to learn robust representations of policy samples. Meanwhile, a synthetic preference generation approach is developed to simulate high-quality preference data with policy outputs. Extensive experiments on three benchmark datasets show that RLP consistently outperforms the state-of-the-art. Our code is available at urlhttps://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/rlp.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 53 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

40 of 53 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
19 more
  1. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023Teaching Large Language Models to Self-Debugcited by 963
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  9. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  12. 2019On Variational Bounds of Mutual Informationcited by 550
  13. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  14. 2022Prompting GPT-3 To Be Reliablecited by 258
  15. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  16. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  17. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  18. 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30
  19. 2023Constructive Large Language Models Alignment with Diverse Feedbackcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
  2. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  3. 2025ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generationcited by 10
  4. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  5. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  6. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedbackcited by 6
  7. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  8. 2025Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planningcited by 3
  9. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlapcited by 3
  10. 2025Navigation-GPT: A Robust and Adaptive Framework Utilizing Large Language Models for Navigation Applicationscited by 0
  11. 2025Context-Emotion Aware Therapeutic Dialogue Generation: A Multi-component Reinforcement Learning Approach to Language Models for Mental Health Supportcited by 0
  12. 2025UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hackingcited by 0
3 more
  1. 2026Imitation Learning for Elder-Facing Speech Synthesiscited by 0
  2. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learningcited by 0
  3. 2026Re2A: Situated Conversational Recommendation via Rubric-based Preference Reasoning and Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  5. 2026The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
  6. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
  7. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  8. 2024Optimal Design for Reward Modeling in RLHF