Data to 13 Oct 2026

Group Robust Preference Optimization in Reward-free RLHF

arXiv 2405.20304 ↗First posted 30 May 2024cs.CL cs.LGcited by 95

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Adapting large language models (LLMs) for specific tasks usually involves fine-tuning through reinforcement learning with human feedback (RLHF) on preference data. While these data often come from diverse labelers' groups (e.g., different demographics, ethnicities, company teams, etc.), traditional RLHF approaches adopt a "one-size-fits-all" approach, i.e., they indiscriminately assume and optimize a single preference model, thus not being robust to unique characteristics and needs of the various groups. To address this limitation, we propose a novel Group Robust Preference Optimization (GRPO) method to align LLMs to individual groups' preferences robustly. Our approach builds upon reward-free direct preference optimization methods, but unlike previous approaches, it seeks a robust policy which maximizes the worst-case group performance. To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss. We theoretically study the feasibility of GRPO and analyze its convergence for the log-linear policy class. By fine-tuning LLMs with GRPO using diverse group-based global opinion data, we significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies compared to non-robust baselines.

Key facts. Cited by 95 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 61 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

45 of 61 references.

Closest ancestors · 4

Same direction, newest first

2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
2022Constitutional AI: Harmlessness from AI Feedback15 Dec
2017Proximal Policy Optimization Algorithms20 Jul

Other references · 41

2022Training language models to follow instructions with human feedbackcited by 15,856
2017Deep reinforcement learning from human preferencescited by 3,871
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
2020Learning to summarize from human feedbackcited by 2,575
2019Fine-Tuning Language Models from Human Preferencescited by 2,139
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,736
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,531
2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,242
2022Scaling Laws for Reward Model Overoptimizationcited by 1,077
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 909
2024Gemma: Open Models Based on Gemini Research and Technologycited by 908
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 751

29 more

2024Self-Rewarding Language Modelscited by 666
2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 648
2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
2023Reinforced Self-Training (ReST) for Language Modelingcited by 492
2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 428
2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 400
2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 391
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 347
2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 337
2023Preference Ranking Optimization for Human Alignmentcited by 284
2023A Survey of Reinforcement Learning from Human Feedbackcited by 261
2024Self-Play Preference Optimization for Language Model Alignmentcited by 229
2023Nash Learning from Human Feedbackcited by 223
2024A Roadmap to Pluralistic Alignmentcited by 179
2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 174
2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 156
2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
2019Distributionally Robust Language Modelingcited by 143
2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 142
2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 63
2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
2024Reinforcement Learning from Human Feedback with Active Queriescited by 40
2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
2024Understanding the Learning Dynamics of Alignment with Human Feedbackcited by 15
2023REDUCR: Robust Data Downsampling Using Class Priority Reweightingcited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 9; 2025: 58; 2026: 28.

95 papers.

Sort:

By area:

2025Dynamic Early Exit in Reasoning Modelscited by 217
2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiencycited by 65
2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 39
2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 38
2025A Survey of Direct Preference Optimizationcited by 35
2025OctoNav: Towards Generalist Embodied Navigationcited by 33
2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
2025Learning to Reason at the Frontier of Learnabilitycited by 23

38 more

2024No Preference Left Behind: Group Distributional Preference Optimizationcited by 21
2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 18
2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 13
2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
2026Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learningcited by 13
2024FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHFcited by 12
2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
2025OpenApps: Simulating Environment Variations to Measure UI-Agent Reliabilitycited by 12
2024Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedbackcited by 10
2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHFcited by 10
2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
2025UniGame: Turning a Unified Multimodal Model Into Its Own Adversarycited by 9
2025Thinking Preference Optimizationcited by 8
2025Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectoriescited by 8
2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
2025A Review of Incorporating Psychological Theories in LLMscited by 5
2025Murphys Laws of AI Alignment: Why the Gap Always Winscited by 5
2025C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMscited by 5
2026FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoningcited by 5
2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoningcited by 5
2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
2025Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equitycited by 4
2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 4
2025Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimizationcited by 4
2025Relative Advantage Debiasing for Watch-Time Prediction in Short-Video Recommendationcited by 4
2025Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Samplingcited by 4
2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 4
2024Strada-LLM: Graph LLM for traffic predictioncited by 3
2025DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Modelscited by 3
2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 3
2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3
2025A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMscited by 3
2025Love First, Know Later: Persona-Based Romantic Compatibility Through LLM Text World Enginescited by 3

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way