Data to 7 Oct 2026

When does return-conditioned supervised learning work for offline reinforcement learning?

arXiv 2206.01079 ↗First posted 2 Jun 2022cs.LGcited by 73

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL). RCSL algorithms learn the distribution of actions conditioned on both the state and the return of the trajectory. Then they define a policy by conditioning on achieving high return. In this paper, we provide a rigorous study of the capabilities and limitations of RCSL, something which is crucially missing in previous work. We find that RCSL returns the optimal policy under a set of assumptions that are stronger than those needed for the more traditional dynamic programming-based algorithms. We provide specific examples of MDPs and datasets that illustrate the necessity of these assumptions and the limits of RCSL. Finally, we present empirical evidence that these limitations will also cause issues in practice by providing illustrative experiments in simple point-mass environments and on datasets from the D4RL benchmark.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 39 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

21 of 39 references.

Closest ancestors · 7

Same direction, newest first

2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
2020Critic Regularized Regression26 Jun
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 14

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2020Scaling Laws for Neural Language Modelscited by 6,411
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2017Hindsight Experience Replaycited by 1,096
2018DeepMind Control Suitecited by 935
2017Constrained Policy Optimizationcited by 928
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
2019Training Agents using Upside-Down Reinforcement Learningcited by 106
2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
2019Reward-Conditioned Policiescited by 101

2 more

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 22; 2024: 22; 2025: 12; 2026: 12.

73 papers.

Sort:

By area:

2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
2025π*0.6: a VLA That Learns From Experiencecited by 280
2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 73
2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58

38 more

2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
2023A Survey on Transformers in Reinforcement Learningcited by 50
2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)cited by 37
2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
2022A Survey on Influence Maximization: From an ML-Based Combinatorial Optimizationcited by 15
2024Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inferencecited by 14
2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
2023CEIL: Generalized Contextual Imitation Learningcited by 12
2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 11
2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learningcited by 10
2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuningcited by 10
2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
2023AlphaStar Unplugged: Large-Scale Offline Reinforcement Learningcited by 8
2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
2023On the Effectiveness of Offline RL for Dialogue Response Generationcited by 6
2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
2024Predictive Coding for Decision Transformercited by 6
2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
2024Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Makingcited by 4
2024Q-value Regularized Decision ConvFormer for Offline Reinforcement Learningcited by 3
2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidancecited by 3

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way