# Reward-Conditioned Policies

arXiv: https://arxiv.org/abs/1912.13465 · HTML: https://mlsanity.com/paper/1912.13465/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2019.

## Built on by (most cited first, 30 of 102)

- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [Foundation Models for Decision Making: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2303.04129/) (2023)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions](https://mlsanity.com/paper/2309.10150/) (2023)
- [Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning](https://mlsanity.com/paper/2304.01203/) (2023)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL](https://mlsanity.com/paper/2209.03993/) (2022)
- [Guided Flows for Generative Modeling and Decision Making](https://mlsanity.com/paper/2311.13443/) (2023)
- [Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone](https://mlsanity.com/paper/2412.06685/) (2024)
- [Can Wikipedia Help Offline Reinforcement Learning?](https://mlsanity.com/paper/2201.12122/) (2022)
- [When does return-conditioned supervised learning work for offline reinforcement learning?](https://mlsanity.com/paper/2206.01079/) (2022)
- [You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments](https://mlsanity.com/paper/2205.15967/) (2022)
- [A Policy-Guided Imitation Approach for Offline Reinforcement Learning](https://mlsanity.com/paper/2210.08323/) (2022)
- [Diffusion Guidance Is a Controllable Policy Improvement Operator](https://mlsanity.com/paper/2505.23458/) (2025)
- [Dichotomy of Control: Separating What You Can Control from What You Cannot](https://mlsanity.com/paper/2210.13435/) (2022)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Beyond Reward: Offline Preference-guided Policy Optimization](https://mlsanity.com/paper/2305.16217/) (2023)
- [Reasoning with Latent Diffusion in Offline Reinforcement Learning](https://mlsanity.com/paper/2309.06599/) (2023)
- [Q-value Regularized Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.17098/) (2024)
- [StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning](https://mlsanity.com/paper/2110.06206/) (2021)
- [Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View](https://mlsanity.com/paper/2401.11237/) (2024)
- [CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning](https://mlsanity.com/paper/2403.19918/) (2024)

## Abstract (arXiv)

Reinforcement learning offers the promise of automating the acquisition of complex behavioral skills. However, compared to commonly used and well-understood supervised learning methods, reinforcement learning algorithms can be brittle, difficult to use and tune, and sensitive to seemingly innocuous implementation decisions. In contrast, imitation learning utilizes standard and well-understood supervised learning methods, but requires near-optimal expert data. Can we learn effective policies via supervised learning without demonstrations? The main idea that we explore in this work is that non-expert trajectories collected from sub-optimal policies can be viewed as optimal supervision, not for maximizing the reward, but for matching the reward of the given trajectory. By then conditioning the policy on the numerical value of the reward, we can obtain a policy that generalizes to larger returns. We show how such an approach can be derived as a principled method for policy search, discuss several variants, and compare the method experimentally to a variety of current reinforcement learning methods on standard benchmarks.