mlsanityData to 7 Oct 2026

Implicit Offline Reinforcement Learning via Supervised Learning

arXiv 2210.12272 ↗First posted 21 Oct 2022stat.ML cs.LG cs.ROcited by 3

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Offline Reinforcement Learning (RL) via Supervised Learning is a simple and effective way to learn robotic skills from a dataset collected by policies of different expertise levels. It is as simple as supervised learning and Behavior Cloning (BC), but takes advantage of return information. On datasets collected by policies of similar expertise, implicit BC has been shown to match or outperform explicit BC. Despite the benefits of using implicit models to learn robotic skills via BC, offline RL via Supervised Learning algorithms have been limited to explicit models. We show how implicit models can leverage return information and match or outperform explicit algorithms to acquire robotic skills from fixed datasets. Furthermore, we show the close relationship between our implicit methods and other popular RL via Supervised Learning algorithms to provide a unified framework. Finally, we demonstrate the effectiveness of our method on high-dimension manipulation and locomotion tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 15 of its 25 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

15 of 25 references.

References · 15

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2021Implicit Behavioral Cloningcited by 442
  11. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  12. 2022Multi-Game Decision Transformerscited by 212
3 more
  1. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  2. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  3. 2019Reward-Conditioned Policiescited by 101

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1; 2026: 1.

3 papers.

  1. 2025Self-evolved Imitation Learning in Simulated Worldcited by 4
  2. 2022Exploring validation metrics for offline model-based optimisation with diffusion modelscited by 1
  3. 2026In-Context Reinforcement Learning through Bayesian Fusion of Context and Value Priorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  2. 2023Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policies
  3. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  4. 2026Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets
  5. 2020Offline Learning from Demonstrations and Unlabeled Experience
  6. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  7. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
  8. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policies