mlsanityData to 7 Oct 2026

Reliable Conditioning of Behavioral Cloning for Offline Reinforcement Learning

arXiv 2210.05158 ↗First posted 11 Oct 2022cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Behavioral cloning (BC) provides a straightforward solution to offline RL by mimicking offline trajectories via supervised learning. Recent advances (Chen et al., 2021; Janner et al., 2021; Emmons et al., 2021) have shown that by conditioning on desired future returns, BC can perform competitively to their value-based counterparts, while enjoying much more simplicity and training stability. While promising, we show that these methods can be unreliable, as their performance may degrade significantly when conditioned on high, out-of-distribution (ood) returns. This is crucial in practice, as we often expect the policy to perform better than the offline dataset by conditioning on an ood value. We show that this unreliability arises from both the suboptimality of training data and model architectures. We propose ConserWeightive Behavioral Cloning (CWBC), a simple and effective method for improving the reliability of conditional BC with two key components: trajectory weighting and conservative regularization. Trajectory weighting upweights the high-return trajectories to reduce the train-test gap for BC methods, while conservative regularizer encourages the policy to stay close to the data distribution for ood conditioning. We study CWBC in the context of RvS (Emmons et al., 2021) and Decision Transformers (Chen et al., 2021), and show that CWBC significantly boosts their performance on various benchmarks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 41 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2022A Generalist Agentcited by 802
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
20 more
  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  5. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  6. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  7. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  8. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  9. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  10. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  11. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  12. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
  13. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  14. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  15. 2019Model Inversion Networks for Model-Based Optimizationcited by 83
  16. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  17. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  18. 2022Generative Pretraining for Black-Box Optimizationcited by 36
  19. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  20. 2021Learning from an Exploring Demonstrator: Optimal Reward Estimation for Banditscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 0; 2025: 0; 2026: 1.

6 papers.

  1. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  2. 2023Diffusion Models for Black-Box Optimizationcited by 75
  3. 2023Diffusion Model-Augmented Behavioral Cloningcited by 37
  4. 2023Aligning Large Language Models with Human: A Surveycited by 31
  5. 2023Decision Stacks: Flexible Reinforcement Learning via Modular Generative Modelscited by 8
  6. 2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
  2. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning
  3. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
  4. 2025Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets
  5. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning
  6. 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning
  7. 2023Improving Behavioural Cloning with Positive Unlabeled Learning
  8. 2024Stable-BC: Controlling Covariate Shift with Stable Behavior Cloning