mlsanityData to 7 Oct 2026

PAFO: Pareto Fairness Optimization for Personalized Reward Modeling

arXiv 2606.07988 ↗First posted 6 Jun 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large language models (LLMs) increasingly rely on reward models to align their outputs with diverse user preferences. While personalized reward models aim to capture such heterogeneity, they are often trained on imbalanced user preference data and may therefore favor users whose preferences are more common in the training population. In this paper, we identify this failure mode as personalized reward bias, where reward modeling quality varies systematically with preference support rate. We formulate its mitigation as a Pareto fairness problem over group utilities, aiming to improve under-served users without degrading other user groups. To this end, we propose PAFO, a Pareto fairness optimization framework for personalized reward modeling. PAFO first trains group-specialized reward models for majority and minority preference groups, then constructs conditional margin-level supervision to distill their heterogeneous preference boundaries into a single unified model. The resulting model uses group information only during training and requires no explicit group labels at inference time. Experiments on Personal-LLM and DSP show that PAFO improves both minority-group and majority-group accuracy while reducing user-level unfairness across multiple metrics, demonstrating its effectiveness for fairer LLM personalization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 53 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

45 of 53 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs5 Jun
  2. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modeling20 Apr
  3. 2025Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models10 Mar
  4. 2025Language Model Personalization via Reward Factorization8 Mar
  5. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning26 Feb
  6. 2024PAD: Personalized Alignment of LLMs at Decoding-Time5 Oct
  7. 2024PersonalLLM: Tailoring LLMs to Individual Preferences30 Sep
  8. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
  9. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  10. 2020Learning to summarize from human feedback2 Sep

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  4. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  5. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  7. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  8. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  9. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  10. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  11. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  12. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
23 more
  1. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  2. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  3. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  4. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  5. 2025Reinforced Latent Reasoning for LLM-based Recommendationcited by 46
  6. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44
  7. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  8. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  9. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  10. 2025NextQuill: Causal Preference Modeling for Enhancing LLM Personalizationcited by 29
  11. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  12. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  13. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximizationcited by 20
  14. 2025Latent Inter-User Difference Modeling for LLM Personalizationcited by 20
  15. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  16. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspectivecited by 9
  17. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  18. 2025SteerX: Disentangled Steering for LLM Personalizationcited by 7
  19. 2024Large Language Models are Biased Reinforcement Learnerscited by 5
  20. 2025Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Expertscited by 4
  21. 2024Fair Recommendations with Limited Sensitive Attributes: A Distributionally Robust Optimization Approachcited by 2
  22. 2026Fairness Aware Reward Optimizationcited by 1
  23. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignmentcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
  2. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  3. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
  4. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization
  5. 2025Detecting Prefix Bias in LLM-based Reward Models
  6. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularization
  7. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
  8. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling