# Learning Relative Return Policies With Upside-Down Reinforcement Learning

arXiv: https://arxiv.org/abs/2202.12742 · HTML: https://mlsanity.com/paper/2202.12742/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 23 Feb 2022.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions](https://mlsanity.com/paper/1912.02875/) (2019)
- [Training Agents using Upside-Down Reinforcement Learning](https://mlsanity.com/paper/1912.02877/) (2019)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Reward-Conditioned Policies](https://mlsanity.com/paper/1912.13465/) (2019)

## Built on by (most cited first, 3 of 3)

- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Upside-Down Reinforcement Learning for More Interpretable Optimal Control](https://mlsanity.com/paper/2411.11457/) (2024)
- [Goal-Conditioned Supervised Learning for LLM Fine-Tuning](https://mlsanity.com/paper/2605.16345/) (2026)

## Abstract (arXiv)

Lately, there has been a resurgence of interest in using supervised learning to solve reinforcement learning problems. Recent work in this area has largely focused on learning command-conditioned policies. We investigate the potential of one such method -- upside-down reinforcement learning -- to work with commands that specify a desired relationship between some scalar value and the observed return. We show that upside-down reinforcement learning can learn to carry out such commands online in a tabular bandit setting and in CartPole with non-linear function approximation. By doing so, we demonstrate the power of this family of methods and open the way for their practical use under more complicated command structures.