mlsanityData to 7 Oct 2026

Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage

arXiv 2106.03207 ↗First posted 6 Jun 2021cs.LG stat.MLcited by 26

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner is presented with a static offline dataset of state-action-next state transition triples from a potentially less proficient behavior policy. We introduce Model-based IL from Offline data (MILO): an algorithmic framework that utilizes the static dataset to solve the offline IL problem efficiently both in theory and in practice. In theory, even if the behavior policy is highly sub-optimal compared to the expert, we show that as long as the data from the behavior policy provides sufficient coverage on the expert state-action traces (and with no necessity for a global coverage over the entire state-action space), MILO can provably combat the covariate shift issue in IL. Complementing our theory results, we also demonstrate that a practical implementation of our approach mitigates covariate shift on benchmark MuJoCo continuous control tasks. We demonstrate that with behavior policies whose performances are less than half of that of the expert, MILO still successfully imitates with an extremely low number of expert state-action pairs while traditional offline IL method such as behavior cloning (BC) fails completely. Source code is provided at https://github.com/jdchang1/milo.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 96 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

48 of 96 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift4 Feb
  2. 2020Toward the Fundamental Limits of Imitation Learning13 Sep
  3. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matching25 Jun
  4. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  5. 2019Provably Efficient Imitation Learning from Observation Alone27 May

Other references · 43

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  11. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  12. 2019Self-Supervised Exploration via Disagreementcited by 263
31 more
  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  3. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  4. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  5. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  6. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  9. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  10. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  11. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  12. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  13. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  14. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  15. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  16. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  17. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  18. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  19. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  20. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  21. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
  22. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  23. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  24. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  25. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  26. 2020Bandit optimisation of functions in the Mat'ern kernel RKHScited by 40
  27. 2019On Online Learning in Kernelized Markov Decision Processescited by 32
  28. 2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27
  29. 2017Goal-Driven Dynamics Learning via Bayesian Optimizationcited by 19
  30. 2020Stable Policy Optimization via Off-Policy Divergence Regularizationcited by 16
  31. 2015Learning to Search for Dependenciescited by 3

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 7; 2023: 4; 2024: 7; 2025: 3.

26 papers.

Sort:

By area:

  1. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  2. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  4. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  5. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  6. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  7. 2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
  8. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  9. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
  10. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  11. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  12. 2023Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behaviorcited by 33
14 more
  1. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
  2. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
  3. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  4. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  5. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 13
  6. 2024SAFE-GIL: SAFEty Guided Imitation Learning for Robotic Systemscited by 13
  7. 2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximationcited by 11
  8. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspectivecited by 4
  9. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approachcited by 3
  10. 2024Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learningcited by 3
  11. 2024Diagnostic Runtime Monitoring with Martingalescited by 2
  12. 2022Optimal Estimation of Off-Policy Policy Gradient via Double Fitted Iterationcited by 1
  13. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Explorationcited by 1
  14. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  2. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories
  3. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon
  4. 2024Offline Imitation Learning with Model-based Reverse Augmentation
  5. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  6. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  7. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching
  8. 2025Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representations