mlsanityData to 7 Oct 2026

Reward Learning through Ranking Mean Squared Error

arXiv 2601.09236 ↗First posted 14 Jan 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward design remains a significant bottleneck in applying reinforcement learning (RL) to real-world problems. A popular alternative is reward learning, where reward functions are inferred from human feedback rather than manually specified. Recent work has proposed learning reward functions from human ratings rather than traditional binary preferences, enabling richer and potentially less cognitively demanding supervision. Building on this paradigm, we introduce a new rating-based RL method, Ranked Return Regression for RL (R4). At its core, R4 uses a novel ranking mean squared error loss that learns from a dataset of trajectory-rating pairs, treating the human-provided discrete ratings (e.g., bad, neutral, good) as ordinal targets. Unlike prior rating-based approaches, R4 offers formal guarantees: its solution set is provably minimal and complete under mild assumptions. Empirically, using both human-provided and simulated ratings, we demonstrate that R4 consistently matches or outperforms existing rating and preference-based RL methods on robotic benchmarks from OpenAI Gym and the DeepMind Control Suite. Code released at https://github.com/IRLL/R4.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 49 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

28 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Reward Learning from Multiple Feedback Types28 Feb
  2. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd19 Nov
  3. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  4. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types23 Aug
  5. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2018DeepMind Control Suitecited by 935
  6. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  7. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  9. 2020Fast Differentiable Sorting and Rankingcited by 196
  10. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  11. 2019Stochastic Optimization of Sorting Networks via Continuous Relaxationscited by 111
  12. 2021Reward (Mis)design for Autonomous Drivingcited by 102
9 more
  1. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  2. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  3. 2022Monotonic Differentiable Sorting Networkscited by 25
  4. 2021Learning Reward Functions from Scale Feedbackcited by 24
  5. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 18
  6. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  7. 2023Rating-based Reinforcement Learningcited by 14
  8. 2025Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitionerscited by 10
  9. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Task Reward Learning from Human Ratings
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  4. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
  5. 2023Is RLHF More Difficult than Standard RL?
  6. 2025Reinforcement Learning from Human Feedback
  7. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  8. 2023Contrastive Preference Learning: Learning from Human Feedback without RL