mlsanityData to 14 Oct 2026

Towards Reward Fairness in RLHF: From a Resource Allocation Perspective

arXiv 2505.23349 ↗First posted 29 May 2025cs.LG cs.AIcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Rewards serve as proxies for human preferences and play a crucial role in Reinforcement Learning from Human Feedback (RLHF). However, if these rewards are inherently imperfect, exhibiting various biases, they can adversely affect the alignment of large language models (LLMs). In this paper, we collectively define the various biases present in rewards as the problem of reward unfairness. We propose a bias-agnostic method to address the issue of reward fairness from a resource allocation perspective, without specifically designing for each type of bias, yet effectively mitigating them. Specifically, we model preference learning as a resource allocation problem, treating rewards as resources to be allocated while considering the trade-off between utility and fairness in their distribution. We propose two methods, Fairness Regularization and Fairness Coefficient, to achieve fairness in rewards. We apply our methods in both verification and reinforcement learning scenarios to obtain a fairness reward model and a policy model, respectively. Experiments conducted in these scenarios demonstrate that our approach aligns LLMs with human preferences in a more fair manner.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 43 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

28 of 43 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  2. 2024WARP: On the Benefits of Weight Averaged Rewarded Policies24 Jun
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  4. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  5. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment2 Feb
  6. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  7. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  8. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  9. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  11. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  12. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
5 more
  1. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  2. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 31
  3. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  4. 2021Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Managementcited by 6
  5. 2024Large Language Models are Biased Reinforcement Learnerscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMscited by 5
  2. 2026Beyond Fixed Psychological Personas: State Beats Trait, but Language Models are State-Blindcited by 4
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  4. 2026Self-Debias: Self-correcting for Debiasing Large Language Modelscited by 3
  5. 2026Linear Social Choice with Few Queries: A Moment-Based Approachcited by 1
  6. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMscited by 1
  7. 2025Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learningcited by 0
  8. 2026ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Taskscited by 0
  9. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Procedural Fairness Failures in RLHF from Preference Averaging
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  3. 2024How to Evaluate Reward Models for RLHF
  4. 2025Detecting Prefix Bias in LLM-based Reward Models
  5. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  6. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
  7. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences
  8. 2026Fairness Aware Reward Optimization