mlsanityData to 7 Oct 2026

Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement

arXiv 2507.06701 ↗First posted 9 Jul 2025cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Imitation Learning from Observation (IfO) offers a powerful way to learn behaviors at large-scale: Unlike behavior cloning or offline reinforcement learning, IfO can leverage action-free demonstrations and thus circumvents the need for costly action-labeled demonstrations or reward functions. However, current IfO research focuses on idealized scenarios with mostly bimodal-quality data distributions, restricting the meaningfulness of the results. In contrast, this paper investigates more nuanced distributions and introduces a method to learn from such data, moving closer to a paradigm in which imitation learning can be performed iteratively via self-improvement. Our method adapts RL-based imitation learning to action-free demonstrations, using a value function to transfer information between expert and non-expert data. Through comprehensive evaluation, we delineate the relation between different data distributions and the applicability of algorithms and highlight the limitations of established methods. Our findings provide valuable insights for developing more robust and practical IfO techniques on a path to scalable behaviour learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 76 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

62 of 76 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasets13 Jun
  2. 2023RLIF: Interactive Imitation Learning as Reinforcement Learning21 Nov
  3. 2023Coherent Soft Imitation Learning25 May
  4. 2022Proximal Point Imitation Learning22 Sep
  5. 2022Improved Policy Optimization for Online Imitation Learning29 Jul
  6. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
  7. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  8. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage6 Jun
  9. 2021Off-Policy Imitation Learning from Observations25 Feb
  10. 2020Offline Learning from Demonstrations and Unlabeled Experience27 Nov
  11. 2020Primal Wasserstein Imitation Learning8 Jun
  12. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
7 more
  1. 2019State Alignment-based Imitation Learning21 Nov
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  3. 2019Imitation Learning as f-Divergence Minimization30 May
  4. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  5. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  6. 2018Generative Adversarial Imitation from Observation17 Jul
  7. 2018Behavioral Cloning from Observation4 May

Other references · 43

  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2016OpenAI Gymcited by 1,684
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2022A Generalist Agentcited by 802
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2022Large Language Models Can Self-Improvecited by 656
  10. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  12. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
31 more
  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  2. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  3. 2016Learning to reinforcement learncited by 369
  4. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  5. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  6. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  7. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  8. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  9. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  10. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  11. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  12. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  13. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  14. 2024Vision Language Models are In-Context Value Learnerscited by 84
  15. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59
  16. 2023Towards A Unified Agent with Foundation Modelscited by 46
  17. 2023Optimal Transport for Offline Imitation Learningcited by 42
  18. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  19. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  20. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  21. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  22. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  23. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  24. 2024Self-Improving Robust Preference Optimizationcited by 19
  25. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  26. 2017Mutual Alignment Transfer Learningcited by 16
  27. 2024Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learningcited by 14
  28. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13
  29. 2023Mastering Stacking of Diverse Shapes with Large-Scale Iterative Reinforcement Learning on Real Robotscited by 13
  30. 2023Massively Scalable Inverse Reinforcement Learning in Google Mapscited by 11
  31. 2023Foundations for Transfer in Reinforcement Learning: A Taxonomy of Knowledge Modalitiescited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2019Recent Advances in Imitation Learning from Observation
  2. 2025Learning from Observation: A Survey of Recent Advances
  3. 2019Imitation Learning from Video by Leveraging Proprioception
  4. 2021DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation
  5. 2023Imitation Learning from Observation with Automatic Discount Scheduling
  6. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances
  7. 2022Imitation Learning from Observations under Transition Model Disparity
  8. 2024Explorative Imitation Learning: A Path Signature Approach for Continuous Environments