mlsanityData to 7 Oct 2026

Reward Collapse in Aligning Large Language Models

arXiv 2305.17608 ↗First posted 28 May 2023cs.LG cs.AI cs.CLcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The extraordinary capabilities of large language models (LLMs) such as ChatGPT and GPT-4 are in part unleashed by aligning them with reward models that are trained on human preferences, which are often represented as rankings of responses to prompts. In this paper, we document the phenomenon of textitreward collapse, an empirical observation where the prevailing ranking-based approach results in an textitidentical reward distribution textitregardless of the prompts during the terminal phase of training. This outcome is undesirable as open-ended prompts like ``write a short story about your best friend'' should yield a continuous range of rewards for their completions, while specific prompts like ``what is the capital of New Zealand'' should generate either high or low rewards. Our theoretical investigation reveals that reward collapse is primarily due to the insufficiency of the ranking-based objective function to incorporate prompt-related information during optimization. This insight allows us to derive closed-form expressions for the reward distribution associated with a set of utility functions in an asymptotic regime. To overcome reward collapse, we introduce a prompt-aware optimization scheme that provably admits a prompt-dependent reward distribution within the interpolating regime. Our experimental results suggest that our proposed prompt-aware utility functions significantly alleviate reward collapse during the training of reward models.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 43 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

21 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Chain of Hindsight Aligns Language Models with Feedback6 Feb
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  7. 2018Learning to Understand Goal Specifications by Modelling Reward5 Jun
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 13

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2021Convex optimizationcited by 2,479
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  12. 2020Calibration of Pre-trained Transformerscited by 307
1 more
  1. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 8; 2025: 7.

23 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  3. 2023Mitigating the Alignment Tax of RLHFcited by 181
  4. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  5. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  6. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  7. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  8. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  9. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  10. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  11. 2025Eliciting Language Model Behaviors with Investigator Agentscited by 21
  12. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20
11 more
  1. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
  2. 2024PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuningcited by 11
  3. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  4. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  5. 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedbackcited by 7
  6. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignmentcited by 4
  7. 2025Adaptive Reinforcement for Open-ended Medical Reasoning via Semantic-Guided Reward Collapse Mitigationcited by 4
  8. 2025GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learningcited by 4
  9. 2023Prevalence and prevention of large language model use in crowd workcited by 1
  10. 2025Tiny Reward Modelscited by 0
  11. 2025SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Robust Reward Modeling for Large Language Models via Causal Decomposition
  2. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
  3. 2024Transforming and Combining Rewards for Aligning Large Language Models
  4. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
  5. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  6. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  7. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  8. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs