mlsanityData to 14 Oct 2026

SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences

arXiv 2509.03672 ↗First posted 3 Sep 2025cs.LG stat.MLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Uniform-reward reinforcement learning from human feedback (RLHF), which trains a single reward model to represent the preferences of all annotators, fails to capture the diversity of opinions across sub-populations, inadvertently favoring dominant groups. The state-of-the-art, MaxMin-RLHF, addresses this by learning group-specific reward models, and by optimizing for the group receiving the minimum reward, thereby promoting fairness. However, we identify that a key limitation of MaxMin-RLHF is its poor performance when the minimum-reward group is a minority. To mitigate this drawback, we introduce a novel framework, termed em SharedRep-RLHF. At its core, SharedRep-RLHF learns and leverages em shared traits in annotations among various groups, in contrast to learning separate reward models across groups. We first show that MaxMin-RLHF is provably suboptimal in learning shared traits, and then quantify the sample complexity of SharedRep-RLHF. Experiments across diverse natural language tasks showcase the effectiveness of SharedRep-RLHF compared to MaxMin-RLHF with a gain of up to 20% in win rate.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 37 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

28 of 37 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Exploration10 Mar
  2. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning19 Aug
  3. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  4. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  5. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  6. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2023Whose Opinions Do Language Models Reflect?cited by 773
  8. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  9. 2024Iterative Reasoning Preference Optimizationcited by 246
  10. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  11. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  12. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
5 more
  1. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 54
  2. 2020Impact of Representation Learning in Linear Banditscited by 52
  3. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  4. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  5. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMscited by 0
  2. 2026Position: The Alignment Community is Unintentionally Building a Censor's Toolkitcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  3. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  4. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHF
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  6. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
  7. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  8. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspective