mlsanityData to 7 Oct 2026

Robust Imitation of a Few Demonstrations with a Backwards Model

arXiv 2210.09337 ↗First posted 17 Oct 2022cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Behavior cloning of expert demonstrations can speed up learning optimal policies in a more sample-efficient way over reinforcement learning. However, the policy cannot extrapolate well to unseen states outside of the demonstration data, creating covariate shift (agent drifting away from demonstrations) and compounding errors. In this work, we tackle this issue by extending the region of attraction around the demonstrations so that the agent can learn how to get back onto the demonstrated trajectories if it veers off-course. We train a generative backwards dynamics model and generate short imagined trajectories from states in the demonstrations. By imitating both demonstrations and these model rollouts, the agent learns the demonstrated paths and how to get back onto these paths. With optimal or near-optimal demonstrations, the learned policy will be both optimal and robust to deviations, with a wider region of attraction. On continuous control domains, we evaluate the robustness when starting from different initial states unseen in the demonstration data. While both our method and other imitation learning baselines can successfully solve the tasks for initial states in the training distribution, our method exhibits considerably more robustness to different initial states.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 31 of its 61 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

31 of 61 references.

References · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  12. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
19 more
  1. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  2. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  3. 2019Planning with Goal-Conditioned Policiescited by 144
  4. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  5. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  6. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  7. 2017Robust Imitation of Diverse Behaviorscited by 85
  8. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  9. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  10. 2020Bidirectional Model-based Policy Optimizationcited by 38
  11. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  12. 2016Funnel Libraries for Real-Time Robust Feedback Motion Planningcited by 28
  13. 2018Recall Traces: Backtracking Models for Efficient Reinforcement Learningcited by 23
  14. 2018Forward-Backward Reinforcement Learningcited by 17
  15. 2021There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learningcited by 17
  16. 2019Generative predecessor models for sample-efficient imitation learningcited by 14
  17. 2019Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Samplingcited by 8
  18. 2020Learning Retrospective Knowledge with Reverse Reinforcement Learningcited by 8
  19. 2018Temporal Regularization in Markov Decision Processcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 1; 2026: 3.

7 papers.

  1. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 34
  2. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
  3. 2023From Past to Future: Rethinking Eligibility Tracescited by 3
  4. 2025Sample-Efficient Behavior Cloning Using General Domain Knowledgecited by 2
  5. 2026GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learningcited by 1
  6. 2026Mitigating State Aliasing in Vision-Language-Action Models via Inverse Dynamics Learningcited by 0
  7. 2026Beyond Conservatism: Recoverability-Conditioned Exploration for Model-Based Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Robust Learning from Observation with Model Misspecification
  2. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics
  3. 2023Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior
  4. 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning
  5. 2025Balance Equation-based Distributionally Robust Offline Imitation Learning
  6. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
  7. 2026Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift
  8. 2022Diverse Imitation Learning via Self-Organizing Generative Models