mlsanityData to 14 Oct 2026

Efficient Imitation under Misspecification

arXiv 2503.13162 ↗First posted 17 Mar 2025cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We consider the problem of imitation learning under misspecification: settings where the learner is fundamentally unable to replicate expert behavior everywhere. This is often true in practice due to differences in observation space and action space expressiveness (e.g. perceptual or morphological differences between robots and humans). Given the learner must make some mistakes in the misspecified setting, interaction with the environment is fundamentally required to figure out which mistakes are particularly costly and lead to compounding errors. However, given the computational cost and safety concerns inherent in interaction, we'd like to perform as little of it as possible while ensuring we've learned a strong policy. Accordingly, prior work has proposed a flavor of efficient inverse reinforcement learning algorithms that merely perform a computationally efficient local search procedure with strong guarantees in the realizable setting. We first prove that under a novel structural condition we term reward-agnostic policy completeness, these sorts of local-search based IRL algorithms are able to avoid compounding errors. We then consider the question of where we should perform local search in the first place, given the learner may not be able to "walk on a tightrope" as well as the expert in the misspecified setting. We prove that in the misspecified setting, it is beneficial to broaden the set of states on which local search is performed to include those reachable by good policies the learner can actually play. We then experimentally explore a variety of sources of misspecification and how offline data can be used to effectively broaden where we perform local search from.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 65 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

43 of 65 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024EvIL: Evolution Strategies for Generalisable Imitation Learning15 Jun
  2. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptions3 May
  3. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning20 Feb
  4. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions11 Jun
  5. 2022Sequence Model Imitation Learning with Unobserved Contexts3 Aug
  6. 2022Minimax Optimal Online Imitation Learning via Replay Estimation30 May
  7. 2022Causal Imitation Learning under Temporally Correlated Noise2 Feb
  8. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Data27 Oct
  9. 2021A Critique of Strictly Batch Imitation Learning5 Oct
  10. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap4 Mar
  11. 2021MobILE: Model-Based Imitation Learning From Observation Alone22 Feb

Other references · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2017Improved Training of Wasserstein GANscited by 2,235
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  9. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  10. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  11. 2019Learning Latent Plans from Playcited by 316
  12. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
20 more
  1. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
  2. 2017Training GANs with Optimismcited by 217
  3. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 189
  4. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  5. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  6. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 130
  7. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  8. 2024WoCoCo: Learning Whole-Body Humanoid Control with Sequential Contactscited by 61
  9. 2022Adapting Rapid Motor Adaptation for Bipedal Robotscited by 54
  10. 2023Learning to Generate Better Than Your LLMcited by 49
  11. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  12. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 36
  13. 2023LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotioncited by 33
  14. 2024Hybrid Inverse Reinforcement Learningcited by 32
  15. 2021Online Apprenticeship Learningcited by 29
  16. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  17. 2023Rapid Motor Adaptation for Robotic Manipulator Armscited by 14
  18. 2023When is Agnostic Reinforcement Learning Statistically Tractable?cited by 9
  19. 2022A Few Expert Queries Suffices for Sample-Efficient RL with Resets and Linear Value Approximationcited by 7
  20. 2024Inverse Reinforcement Learning with Sub-optimal Expertscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 17
  2. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPscited by 5
  3. 2026When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizoncited by 3
  4. 2026Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shiftcited by 0
  5. 2026A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Misspecification in Inverse Reinforcement Learning
  2. 2022Deconfounding Imitation Learning with Variational Inference
  3. 2026When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning
  4. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification
  5. 2022Imitation Learning by Estimating Expertise of Demonstrators
  6. 2021Learning Online from Corrective Feedback: A Meta-Algorithm for Robotics
  7. 2019Causal Confusion in Imitation Learning
  8. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges