mlsanityData to 7 Oct 2026

RvS: What is Essential for Offline RL via Supervised Learning?

arXiv 2112.10751 ↗First posted 20 Dec 2021cs.LG cs.AI stat.MLcited by 184

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. When does this hold true, and which algorithmic components are necessary? Through extensive experiments, we boil supervised learning for offline RL down to its essential elements. In every environment suite we consider, simply maximizing likelihood with a two-layer feedforward MLP is competitive with state-of-the-art results of substantially more complex methods based on TD learning or sequence modeling with Transformers. Carefully choosing model capacity (e.g., via regularization or architecture) and choosing which information to condition on (e.g., goals or rewards) are critical for performance. These insights serve as a field guide for practitioners doing Reinforcement Learning via Supervised Learning (which we coin "RvS learning"). They also probe the limits of existing RvS methods, which are comparatively weak on random data, and suggest a number of open problems.

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 43 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

36 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2021Offline RL Without Off-Policy Evaluation16 Jun
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  4. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  5. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  10. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  11. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  12. 2019On Mutual Information Maximization for Representation Learningcited by 315
15 more
  1. 2019Learning Latent Plans from Playcited by 310
  2. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  3. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  4. 2019Goal-conditioned Imitation Learningcited by 159
  5. 2020Model-Based Offline Planningcited by 126
  6. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  7. 2021Scaling Laws for Neural Machine Translationcited by 110
  8. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  9. 2019Reward-Conditioned Policiescited by 101
  10. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  11. 2019Urban Driving with Conditional Imitation Learningcited by 84
  12. 2019ROBEL: Robotics Benchmarks for Learning with Low-Cost Robotscited by 70
  13. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  14. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 53
  15. 2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 28; 2023: 58; 2024: 51; 2025: 27; 2026: 20.

184 papers.

Sort:

By area:

  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  2. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  3. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  4. 2023Pretraining Language Models with Human Preferencescited by 245
  5. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  6. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  7. 2022Online Decision Transformercited by 194
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  9. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  10. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  11. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  12. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
38 more
  1. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  2. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  3. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  4. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  5. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  6. 2023OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensemblingcited by 71
  7. 2024Simple Hierarchical Planning with Diffusioncited by 71
  8. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
  9. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  10. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  11. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  12. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  13. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  14. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  15. 2023Behavior Proximal Policy Optimizationcited by 43
  16. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  17. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  18. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 40
  19. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  20. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  21. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  22. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  23. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  24. 2023Contrastive Difference Predictive Codingcited by 35
  25. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  26. 2024Q-value Regularized Transformer for Offline Reinforcement Learningcited by 28
  27. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 27
  28. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  29. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  30. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  31. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 26
  32. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 25
  33. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  34. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  35. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 23
  36. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  37. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learningcited by 22
  38. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22

50 most cited of 184 All 184 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  2. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  3. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  4. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
  5. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  6. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  7. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  8. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning