# Rating-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2307.16348 · HTML: https://mlsanity.com/paper/2307.16348/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 30 Jul 2023.

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [B-Pref: Benchmarking Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2111.03026/) (2021)
- [SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning](https://mlsanity.com/paper/2203.10050/) (2022)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [Preference-based Reinforcement Learning with Finite-Time Guarantees](https://mlsanity.com/paper/2006.08910/) (2020)
- [Reward Uncertainty for Exploration in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2205.12401/) (2022)
- [APReL: A Library for Active Preference-based Reward Learning Algorithms](https://mlsanity.com/paper/2108.07259/) (2021)
- [Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach](https://mlsanity.com/paper/2010.07467/) (2020)
- [BC-IRL: Learning Generalizable Reward Functions from Demonstrations](https://mlsanity.com/paper/2303.16194/) (2023)

## Built on by (most cited first, 15 of 15)

- [Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback](https://mlsanity.com/paper/2402.02423/) (2024)
- [Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models](https://mlsanity.com/paper/2506.12822/) (2025)
- [Listwise Reward Estimation for Offline Preference-based Reinforcement Learning](https://mlsanity.com/paper/2408.04190/) (2024)
- [Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2405.00746/) (2024)
- [Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences](https://mlsanity.com/paper/2409.07268/) (2024)
- [Multi-Task Reward Learning from Human Ratings](https://mlsanity.com/paper/2506.09183/) (2025)
- [MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics](https://mlsanity.com/paper/2603.09237/) (2026)
- [Human-Inspired Multi-Level Reinforcement Learning](https://mlsanity.com/paper/2501.07502/) (2025)
- [Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning](https://mlsanity.com/paper/2602.04518/) (2026)
- [AirSplat: Alignment and Rating for Robust Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2603.25129/) (2026)
- [Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity](https://mlsanity.com/paper/2406.06495/) (2024)
- [Reward Learning through Ranking Mean Squared Error](https://mlsanity.com/paper/2601.09236/) (2026)
- [Distributed Direct Preference Optimization](https://mlsanity.com/paper/2605.20696/) (2026)
- [CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning](https://mlsanity.com/paper/2607.01721/) (2026)
- [Action- and Language-Conditioned Video Assessment for Embodied Control](https://mlsanity.com/paper/2608.08273/) (2026)

## Abstract (arXiv)

This paper develops a novel rating-based reinforcement learning approach that uses human ratings to obtain human guidance in reinforcement learning. Different from the existing preference-based and ranking-based reinforcement learning paradigms, based on human relative preferences over sample pairs, the proposed rating-based reinforcement learning approach is based on human evaluation of individual trajectories without relative comparisons between sample pairs. The rating-based reinforcement learning approach builds on a new prediction model for human ratings and a novel multi-class loss function. We conduct several experimental studies based on synthetic ratings and real human ratings to evaluate the effectiveness and benefits of the new rating-based reinforcement learning approach.