mlsanityData to 7 Oct 2026

Off-Policy Imitation Learning from Observations

arXiv 2102.13185 ↗First posted 25 Feb 2021cs.LG cs.AIcited by 70

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Learning from Observations (LfO) is a practical reinforcement learning scenario from which many applications can benefit through the reuse of incomplete resources. Compared to conventional imitation learning (IL), LfO is more challenging because of the lack of expert action guidance. In both conventional IL and LfO, distribution matching is at the heart of their foundation. Traditional distribution matching approaches are sample-costly which depend on on-policy transitions for policy learning. Towards sample-efficiency, some off-policy solutions have been proposed, which, however, either lack comprehensive theoretical justifications or depend on the guidance of expert actions. In this work, we propose a sample-efficient LfO approach that enables off-policy optimization in a principled manner. To further accelerate the learning procedure, we regulate the policy update with an inverse action model, which assists distribution matching from the perspective of mode-covering. Extensive empirical results on challenging locomotion tasks indicate that our approach is comparable with state-of-the-art in terms of both sample-efficiency and asymptotic performance.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 45 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

27 of 45 references.

Closest ancestors · 12

Same direction, newest first

  1. 2020State-only Imitation with Transition Dynamics Mismatch27 Feb
  2. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  3. 2019State Alignment-based Imitation Learning21 Nov
  4. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
  5. 2019Wasserstein Adversarial Imitation Learning19 Jun
  6. 2019Recent Advances in Imitation Learning from Observation30 May
  7. 2019Provably Efficient Imitation Learning from Observation Alone27 May
  8. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow1 Oct
  9. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  10. 2018Generative Adversarial Imitation from Observation17 Jul
  11. 2018Imitating Latent Policies from Observation21 May
  12. 2018Behavioral Cloning from Observation4 May

Other references · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2017Wasserstein GANcited by 635
  6. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  11. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  12. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
3 more
  1. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  2. 2017Third-Person Imitation Learningcited by 103
  3. 2018Playing hard exploration games by watching YouTubecited by 98

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 15; 2023: 21; 2024: 16; 2025: 10; 2026: 3.

70 papers.

Sort:

By area:

  1. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  2. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  3. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  4. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  5. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  6. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  7. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  8. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  9. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  10. 2022Planning for Sample Efficient Imitation Learningcited by 20
  11. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  12. 2022A Ranking Game for Imitation Learningcited by 18
38 more
  1. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  2. 2023DiffAIL: Diffusion Adversarial Imitation Learningcited by 18
  3. 2023Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challengescited by 17
  4. 2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
  5. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matchingcited by 15
  6. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14
  7. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  8. 2021Imitation by Predicting Observationscited by 13
  9. 2022Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimizationcited by 12
  10. 2023A Coupled Flow Approach to Imitation Learningcited by 12
  11. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  12. 2024Diffusion Imitation from Observationcited by 12
  13. 2022Imitation Learning from Observations under Transition Model Disparitycited by 11
  14. 2022Adversarial Imitation Learning from Video using a State Observercited by 10
  15. 2023Adversarial Imitation Learning from Visual Observations using Latent Informationcited by 10
  16. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  17. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  18. 2023Vision-based Learning for Drones: A Surveycited by 9
  19. 2024Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learningcited by 9
  20. 2023Imitation Learning from Observation through Optimal Transportcited by 7
  21. 2022Imitation Learning by State-Only Distribution Matchingcited by 6
  22. 2023Learning non-Markovian Decision-Making from State-only Sequencescited by 6
  23. 2023Mimicking Better by Matching the Approximate Action Distributioncited by 5
  24. 2023Offline Imitation from Observation via Primal Wasserstein State Occupancy Matchingcited by 5
  25. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 5
  26. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  27. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  28. 2025Controlling Large Language Model with Latent Actionscited by 5
  29. 2021DEALIO: Data-Efficient Adversarial Learning for Imitation from Observationcited by 4
  30. 2024Offline Hierarchical Reinforcement Learning via Inverse Optimizationcited by 4
  31. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  32. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  33. 2024Fusion Dynamical Systems with Machine Learning in Imitation Learning: A Comprehensive Overviewcited by 3
  34. 2024Explorative Imitation Learning: A Path Signature Approach for Continuous Environmentscited by 3
  35. 2024How to Choose a Reinforcement-Learning Algorithmcited by 3
  36. 2024Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learningcited by 3
  37. 2021RAIL: A modular framework for Reinforcement-learning-based Adversarial Imitation Learningcited by 2
  38. 2022Imitation Learning via Differentiable Physicscited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020On the Guaranteed Almost Equivalence between Imitation Learning from Observation and Demonstration
  2. 2026Imitation from Observations with Trajectory-Level Generative Embeddings
  3. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  4. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  5. 2023Imitation Learning from Observation with Automatic Discount Scheduling
  6. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  7. 2024Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learning
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism