# ELO-Rated Sequence Rewards: Advancing Reinforcement Learning Models

arXiv: https://arxiv.org/abs/2409.03301 · HTML: https://mlsanity.com/paper/2409.03301/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping](https://mlsanity.com/paper/2011.02669/) (2020)
- [B-Pref: Benchmarking Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2111.03026/) (2021)
- [Reinforcement Learning with Trajectory Feedback](https://mlsanity.com/paper/2008.06036/) (2020)
- [Learning Long-Term Reward Redistribution via Randomized Return Decomposition](https://mlsanity.com/paper/2111.13485/) (2021)
- [Synthetic Returns for Long-Term Credit Assignment](https://mlsanity.com/paper/2102.12425/) (2021)

## Built on by (most cited first, 1 of 1)

- [Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings](https://mlsanity.com/paper/2506.00178/) (2025)

## Abstract (arXiv)

Reinforcement Learning (RL) heavily relies on the careful design of the reward function. However, accurately assigning rewards to each state-action pair in Long-Term Reinforcement Learning (LTRL) tasks remains a significant challenge. As a result, RL agents are often trained under expert guidance. Inspired by the ordinal utility theory in economics, we propose a novel reward estimation algorithm: ELO-Rating based Reinforcement Learning (ERRL). This approach features two key contributions. First, it uses expert preferences over trajectories rather than cardinal rewards (utilities) to compute the ELO rating of each trajectory as its reward. Second, a new reward redistribution algorithm is introduced to alleviate training instability in the absence of a fixed anchor reward. In long-term scenarios (up to 5000 steps), where traditional RL algorithms struggle, our method outperforms several state-of-the-art baselines. Additionally, we conduct a comprehensive analysis of how expert preferences influence the results.