# Multi-Task Reward Learning from Human Ratings

arXiv: https://arxiv.org/abs/2506.09183 · HTML: https://mlsanity.com/paper/2506.09183/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Rating-based Reinforcement Learning](https://mlsanity.com/paper/2307.16348/) (2023)
- [Human-Inspired Multi-Level Reinforcement Learning](https://mlsanity.com/paper/2501.07502/) (2025)
- [Performance Optimization of Ratings-Based Reinforcement Learning](https://mlsanity.com/paper/2501.07755/) (2025)

## Built on by (most cited first, 2 of 2)

- [Reward-free Alignment for Conflicting Objectives](https://mlsanity.com/paper/2602.02495/) (2026)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)

## Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has become a key factor in aligning model behavior with users' goals. However, while humans integrate multiple strategies when making decisions, current RLHF approaches often simplify this process by modeling human reasoning through isolated tasks such as classification or regression. In this paper, we propose a novel reinforcement learning (RL) method that mimics human decision-making by jointly considering multiple tasks. Specifically, we leverage human ratings in reward-free environments to infer a reward function, introducing learnable weights that balance the contributions of both classification and regression models. This design captures the inherent uncertainty in human decision-making and allows the model to adaptively emphasize different strategies. We conduct several experiments using synthetic human ratings to validate the effectiveness of the proposed approach. Results show that our method consistently outperforms existing rating-based RL methods, and in some cases, even surpasses traditional RL approaches.