mlsanityData to 7 Oct 2026

Imitating Past Successes can be Very Suboptimal

arXiv 2206.03378 ↗First posted 7 Jun 2022cs.LG cs.AIcited by 20

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Prior work has proposed a simple strategy for reinforcement learning (RL): label experience with the outcomes achieved in that experience, and then imitate the relabeled experience. These outcome-conditioned imitation learning methods are appealing because of their simplicity, strong performance, and close ties with supervised learning. However, it remains unclear how these methods relate to the standard RL objective, reward maximization. In this paper, we formally relate outcome-conditioned imitation learning to reward maximization, drawing a precise relationship between the learned policy and Q-values and explaining the close connections between these methods and prior EM-based policy search methods. This analysis shows that existing outcome-conditioned imitation learning methods do not necessarily improve the policy, but a simple modification results in a method that does guarantee policy improvement, under some assumptions.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 44 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

21 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Goal-conditioned Imitation Learning13 Jun
  2. 2018Self-Imitation Learning14 Jun

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2017Deep Reinforcement Learning that Matterscited by 727
  6. 2019A Theory of Regularized Markov Decision Processescited by 279
  7. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  8. 2018Semi-parametric Topological Memory for Navigationcited by 203
  9. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  10. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  11. 2019Reward-Conditioned Policiescited by 101
  12. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
7 more
  1. 2016Multi-Objective Deep Reinforcement Learningcited by 81
  2. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  3. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  4. 2020Planning from Pixels using Inverse Dynamics Modelscited by 36
  5. 2021Interactive Learning from Activity Descriptioncited by 26
  6. 2019Policy Continuation with Hindsight Inverse Dynamicscited by 22
  7. 2021MHER: Model-based Hindsight Experience Replaycited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 2; 2025: 5; 2026: 3.

20 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  2. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  3. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  4. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  5. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  6. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  7. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  8. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  9. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  10. 2024Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inferencecited by 14
  11. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 13
  12. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
8 more
  1. 2023LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Followingcited by 8
  2. 2025GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learningcited by 2
  3. 2026Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Successcited by 2
  4. 2025How to Provably Improve Return Conditioned Supervised Learning?cited by 1
  5. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
  6. 2026Unified Visuomotor Targets: Supervising VLAs Beyond Physical Actionscited by 1
  7. 2023Efficiency Separation between RL Methods: Model-Free, Model-Based and Goal-Conditionedcited by 0
  8. 2025Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022A Ranking Game for Imitation Learning
  2. 2022Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning
  3. 2023Residual Q-Learning: Offline and Online Policy Customization without Value
  4. 2022STIR2: Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward tasks
  5. 2021Imitation Learning by Reinforcement Learning
  6. 2018Hierarchical Imitation and Reinforcement Learning
  7. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges
  8. 2019Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations