mlsanityData to 7 Oct 2026

General Preference Reinforcement Learning

arXiv 2605.18721 ↗First posted 18 May 2026cs.LG cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Post-training has split large language model (LLM) alignment into two largely disconnected tracks. Online reinforcement learning (RL) with verifiable rewards drives emergent reasoning on math and code but depends on a programmatic verifier that cannot reach open-ended tasks, while preference optimization handles open-ended generation yet forgoes the continuous exploration that powers online RL. Closing this gap requires a verifier for open-ended quality, but a scalar reward model is the wrong shape for the job. Quality is multi-dimensional, and any scalar score is an incomplete proxy that lets online RL collapse onto whichever axis the score is most sensitive to. We turn instead to the General Preference Model (GPM), which embeds responses into k skew-symmetric subspaces and represents preference as a structured, intransitivity-aware comparison. Building on this, we propose General Preference Reinforcement Learning (GPRL), which carries the k-way structure through to the policy update. GPRL computes per-dimension group-relative advantages, normalizes each on its own scale so no axis can dominate, and aggregates them with context-dependent eigenvalues. The same structure powers a closed-loop drift monitor that detects single-axis exploitation and corrects it on the fly by reweighting dimensions and tightening the trust region. Starting from textttLlama-3-8B-Instruct, GPRL reaches a length-controlled win rate of 56.51% on AlpacaEval~2.0 while also outperforming SimPO and SPPO on Arena-Hard, MT-Bench, and WildBench by resisting reward hacking across extended training runs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 53 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

44 of 53 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Reward Shaping to Mitigate Reward Hacking in RLHF26 Feb
  2. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF21 Feb
  3. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment16 Jan
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  5. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment3 Oct
  6. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification19 Jul
  7. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  8. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  9. 2023Nash Learning from Human Feedback1 Dec
  10. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
2 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2024A Survey on LLM-as-a-Judgecited by 1,384
  11. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
18 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  4. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  5. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  6. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  7. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  8. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  9. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  10. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  11. 2021Consequences of Misaligned AIcited by 79
  12. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  13. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  14. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  15. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  16. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Modelcited by 25
  17. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  18. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressurecited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration
  2. 2025Bridging Offline and Online Reinforcement Learning for LLMs
  3. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
  4. 2025ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning
  5. 2025Online Preference Alignment for Language Models via Count-based Exploration
  6. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  7. 2026GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zero
  8. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization