mlsanityData to 14 Oct 2026

Training Agents using Upside-Down Reinforcement Learning

arXiv 1912.02877 ↗First posted 5 Dec 2019cs.LG cs.AI cs.ROcited by 107

Not on the map. Why?

Cited by 52 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We develop Upside-Down Reinforcement Learning (UDRL), a method for learning to act using only supervised learning techniques. Unlike traditional algorithms, UDRL does not use reward prediction or search for an optimal policy. Instead, it trains agents to follow commands such as "obtain so much total reward in so much time." Many of its general principles are outlined in a companion report; the goal of this paper is to develop a practical learning algorithm and show that this conceptually simple perspective on agent training can produce a range of rewarding behaviors for multiple episodic environments. Experiments show that on some tasks UDRL's performance can be surprisingly competitive with, and even exceed that of some traditional baseline algorithms developed over decades of research. Based on these results, we suggest that alternative approaches to expected reward maximization have an important role to play in training useful autonomous agents.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 25; 2023: 39; 2024: 17; 2025: 11; 2026: 4.

107 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  4. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  5. 2022Multi-Game Decision Transformerscited by 214
  6. 2022Online Decision Transformercited by 196
  7. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  8. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  9. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
  10. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  11. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  12. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
38 more
  1. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  2. 2023Guided Flows for Generative Modeling and Decision Makingcited by 101
  3. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  4. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  5. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  6. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  7. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  8. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 59
  9. 2023Anti-Exploration by Random Network Distillationcited by 50
  10. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  11. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 42
  12. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  13. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  14. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  15. 2023Contrastive Difference Predictive Codingcited by 36
  16. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  17. 2024The broader spectrum of in-context learningcited by 35
  18. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  19. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learningcited by 33
  20. 2024Q-value Regularized Transformer for Offline Reinforcement Learningcited by 29
  21. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  22. 2023Learning to Modulate pre-trained Models in RLcited by 28
  23. 2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
  24. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 27
  25. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 26
  26. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  27. 2022Imitating Past Successes can be Very Suboptimalcited by 23
  28. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 23
  29. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  30. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
  31. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  32. 2023Passive learning of active causal strategies in agents and language modelscited by 21
  33. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  34. 2023Graph Decision Transformercited by 20
  35. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?cited by 20
  36. 2022Hypernetworks for Zero-shot Transfer in Reinforcement Learningcited by 19
  37. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  38. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18

50 most cited of 107 All 107 (JSON)