Data to 7 Oct 2026

On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

arXiv 2405.16455 ↗First posted 26 May 2024stat.ML cs.LG stat.MEcited by 72

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Accurately aligning large language models (LLMs) with human preferences is crucial for informing fair, economically sound, and statistically efficient decision-making processes. However, we argue that the predominant approach for aligning LLMs with human preferences through a reward model -- reinforcement learning from human feedback (RLHF) -- suffers from an inherent algorithmic bias due to its Kullback--Leibler-based regularization in optimization. In extreme cases, this bias could lead to a phenomenon we term preference collapse, where minority preferences are virtually disregarded. To mitigate this algorithmic bias, we introduce preference matching (PM) RLHF, a novel approach that provably aligns LLMs with the preference distribution of the reward model under the Bradley--Terry--Luce/Plackett--Luce model. Central to our approach is a PM regularizer that takes the form of the negative logarithm of the LLM's policy probability distribution over responses, which helps the LLM balance response diversification and reward maximization. Notably, we obtain this regularizer by solving an ordinary differential equation that is necessary for the PM property. For practical implementation, we introduce a conditional variant of PM RLHF that is tailored to natural language generation. Finally, we empirically validate the effectiveness of conditional PM RLHF through experiments on the OPT and Llama-family models, demonstrating a 29% to 41% improvement in alignment with human preferences, as measured by a certain metric, compared to standard RLHF.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 74 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

45 of 74 references.

Closest ancestors · 11

Same direction, newest first

Other references · 34

2017Attention Is All You Needcited by 59,932
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2020Language Models are Few-Shot Learnerscited by 32,199
2023GPT-4 Technical Reportcited by 18,132
2017Proximal Policy Optimization Algorithmscited by 14,441
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
2019The Curious Case of Neural Text Degenerationcited by 2,881

22 more

2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
2020Learning to summarize from human feedbackcited by 2,562
2023Whose Opinions Do Language Models Reflect?cited by 759
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
2023Nash Learning from Human Feedbackcited by 220
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 160
2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 51
2024Dataset Reset Policy Optimization for RLHFcited by 41
2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 31
2023Reward Collapse in Aligning Large Language Modelscited by 23
2023RLHF and IIA: Perverse Incentivescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 31; 2026: 35.

72 papers.

Sort:

By area:

2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
2024Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactionscited by 50
2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
2024Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modelingcited by 20
2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 19
2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18

38 more

2024Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmeticcited by 14
2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
2025Doubly Robust Alignment for Large Language Modelscited by 11
2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matchingcited by 10
2025Silenced Biases: The Dark Side LLMs Learned to Refusecited by 10
2026Where does output diversity collapse in post-training?cited by 10
2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 9
2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 9
2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedbackcited by 7
2025An Empirical Study of Group Conformity in Multi-Agent Systemscited by 7
2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 6
2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
2026Beyond Marginal Distributions: A Framework to Evaluate the Representativeness of Demographic-Aligned LLMscited by 4
2026CATTO: Balancing Preferences and Confidence in Language Modelscited by 4
2026Alignment Makes Language Models Normative, Not Descriptivecited by 4
2026Steering LLMs for Culturally Localized Generationcited by 4
2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3
2025A note on conditional PAC-efficient reasoning in large language model routingcited by 3
2026Reward Models Inherit Value Biases from Pretrainingcited by 3
2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantificationcited by 3
2025Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Modelscited by 2
2025Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Textscited by 2
2025The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curationcited by 2
2026Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implicationscited by 2
2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuningcited by 2
2026Probabilistic Calibration Is a Trainable Capability in Language Modelscited by 2
2026AI-Associated Lexical Shifts Across 34 Languages: Cross-Lingual Convergence and Diachronic Uptake in News Writingcited by 2
2025Let's Unlearn Stereotypes Before Decision-Making: Assessing the Impact of Intrinsic Bias Mitigation on Downstream Fairness in LLMscited by 1
2025Minority-Aware Satisfaction Estimation in Dialogue Systems via Preference-Adaptive Reinforcement Learningcited by 1
2025Making LLMs Reliable When It Matters Most: A Five-Layer Architecture for High-Stakes Decisionscited by 1
2026The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solvingcited by 1
2026VISPA: Pluralistic Alignment via Automatic Value Selection and Activationcited by 1
2026Not All Layers Need Tuning: Selective Layer Restoration Recovers Diversitycited by 1

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way