mlsanityData to 7 Oct 2026

Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards

arXiv 2403.07708 ↗First posted 12 Mar 2024cs.CL cs.AIcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is the mainstream paradigm used to align large language models (LLMs) with human preferences. Yet existing RLHF heavily relies on accurate and informative reward models, which are vulnerable and sensitive to noise from various sources, e.g. human labeling errors, making the pipeline fragile. In this work, we improve the effectiveness of the reward model by introducing a penalty term on the reward, named as textitcontrastive rewards. %Contrastive rewards Our approach involves two steps: (1) an offline sampling step to obtain responses to prompts that serve as baseline calculation and (2) a contrastive reward calculated using the baseline responses and used in the Proximal Policy Optimization (PPO) step. We show that contrastive rewards enable the LLM to penalize reward uncertainty, improve robustness, encourage improvement over baselines, calibrate according to task difficulty, and reduce variance in PPO. We show empirically contrastive rewards can improve RLHF substantially, evaluated by both GPTs and humans, and our method consistently outperforms strong baselines.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 44 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

26 of 44 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Data17 Dec
  2. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  3. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  5. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  6. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  10. 2018Scalable agent alignment via reward modeling: a research direction19 Nov

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  10. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  11. 2021Alignment of Language Agentscited by 167
  12. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
4 more
  1. 2021Clusterability as an Alternative to Anchor Points When Learning with Noisy Labelscited by 65
  2. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  3. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  4. 2020Policy Learning Using Weak Supervisioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 1.

23 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  3. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  4. 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 48
  5. 2024Reward-Robust RLHF in LLMscited by 32
  6. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  7. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actionscited by 15
  8. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  9. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  10. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  11. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  12. 2025ENCORE: Entropy-guided Reward Composition for Multi-head Safety Reward Modelscited by 4
11 more
  1. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  2. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  3. 2025GCPO: When Contrast Fails, Go Goldcited by 2
  4. 2025Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inferencecited by 2
  5. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  6. 2025Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignmentcited by 1
  7. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  8. 2025Reasoning without Regretcited by 0
  9. 2025ACPO: Adaptive Curriculum Policy Optimization for Aligning Vision-Language Models in Complex Reasoningcited by 0
  10. 2025Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedbackcited by 0
  11. 2025PC-Diffusion: Aligning Diffusion Models with Human Preferences via Preference Classifiercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF
  2. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution
  6. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  7. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  8. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning