mlsanityData to 7 Oct 2026

All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL

arXiv 2202.11960 ↗First posted 24 Feb 2022cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Upside down reinforcement learning (UDRL) flips the conventional use of the return in the objective function in RL upside down, by taking returns as input and predicting actions. UDRL is based purely on supervised learning, and bypasses some prominent issues in RL: bootstrapping, off-policy corrections, and discount factors. While previous work with UDRL demonstrated it in a traditional online RL setting, here we show that this single algorithm can also work in the imitation learning and offline RL settings, be extended to the goal-conditioned RL setting, and even the meta-RL setting. With a general agent architecture, a single UDRL agent can learn across all paradigms.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 38 references that are in the map. First posted on arXiv 24 Feb 2022.

Builds on

21 of 38 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  6. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  7. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  8. 2016Learning to reinforcement learncited by 369
  9. 2019Causal Confusion in Imitation Learningcited by 286
  10. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  11. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  12. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
9 more
  1. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  2. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  3. 2017Third-Person Imitation Learningcited by 103
  4. 2019Reward-Conditioned Policiescited by 101
  5. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  6. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  7. 2021Hierarchical Reinforcement Learning with Timed Subgoalscited by 20
  8. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  9. 2020On Catastrophic Interference in Atari 2600 Gamescited by 13

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 1; 2025: 2.

9 papers.

  1. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  2. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  3. 2023Passive learning of active causal strategies in agents and language modelscited by 21
  4. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  5. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  6. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  7. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  8. 2025Action Chunking with Transformers for Image-Based Spacecraft Guidance and Controlcited by 2
  9. 2022Enhanced Meta Reinforcement Learning using Demonstrations in Sparse Reward Environmentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
  2. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
  3. 2024Hybrid Inverse Reinforcement Learning
  4. 2023Inverse Reinforcement Learning without Reinforcement Learning
  5. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  6. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning
  7. 2023Curricular Subgoals for Inverse Reinforcement Learning
  8. 2024Unsupervised-to-Online Reinforcement Learning