mlsanityData to 7 Oct 2026

Feedback in Imitation Learning: The Three Regimes of Covariate Shift

arXiv 2102.02872 ↗First posted 4 Feb 2021cs.LG cs.RO stat.MLcited by 68

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Imitation learning practitioners have often noted that conditioning policies on previous actions leads to a dramatic divergence between "held out" error and performance of the learner in situ. Interactive approaches can provably address this divergence but require repeated querying of a demonstrator. Recent work identifies this divergence as stemming from a "causal confound" in predicting the current action, and seek to ablate causal aspects of current state using tools from causal inference. In this work, we argue instead that this divergence is simply another manifestation of covariate shift, exacerbated particularly by settings of feedback between decisions and input features. The learner often comes to rely on features that are strongly predictive of decisions, but are subject to strong covariate shift. Our work demonstrates a broad class of problems where this shift can be mitigated, both theoretically and practically, by taking advantage of a simulator but without any further querying of expert demonstration. We analyze existing benchmarks used to test imitation learning approaches and find that these benchmarks are realizable and simple and thus insufficient for capturing the harder regimes of error compounding seen in real-world decision making problems. We find, in a surprising contrast with previous literature, but consistent with our theory, that naive behavioral cloning provides excellent results. We detail the need for new standardized benchmarks that capture the phenomena seen in robotics problems.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 11 of its 31 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

11 of 31 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Histories28 Oct
  2. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization23 Jun
  3. 2019Causal Confusion in Imitation Learning28 May
  4. 2019Provably Efficient Imitation Learning from Observation Alone27 May

Other references · 7

  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2016End to End Learning for Self-Driving Carscited by 1,000
  3. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  4. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  5. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  6. 2017Imitating Driver Behavior with Generative Adversarial Networkscited by 92
  7. 2016Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrationscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 13; 2023: 16; 2024: 10; 2025: 17; 2026: 7.

68 papers.

Sort:

By area:

  1. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  2. 2023Data Quality in Imitation Learningcited by 72
  3. 2023HYDRA: Hybrid Robot Actions for Imitation Learningcited by 54
  4. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  5. 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 38
  6. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  7. 2025ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Contextcited by 34
  8. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  9. 2022Causal Imitation Learning under Temporally Correlated Noisecited by 28
  10. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  11. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  12. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
38 more
  1. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  2. 2024Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Modelscited by 22
  3. 2025RoboCopilot: Human-in-the-loop Interactive Imitation Learning for Robot Manipulationcited by 22
  4. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problemcited by 18
  5. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  6. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  7. 2022Fighting Fire with Fire: Avoiding DNN Shortcuts through Primingcited by 15
  8. 2022Resolving Copycat Problems in Visual Imitation Learning via Residual Action Predictioncited by 14
  9. 2024FutureDepth: Learning to Predict the Future Improves Video Depth Estimationcited by 13
  10. 2025Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecificationcited by 12
  11. 2024Efficient Imitation Learning with Conservative World Modelscited by 11
  12. 2024Multi-Agent Imitation Learning: Value is Easy, Regret is Hardcited by 11
  13. 2024Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedbackcited by 11
  14. 2022Receding Horizon Inverse Reinforcement Learningcited by 10
  15. 2022Visual Backtracking Teleoperation: A Data Collection Protocol for Offline Image-Based Reinforcement Learningcited by 10
  16. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  17. 2022Characterizing the robustness of Bayesian adaptive experimental designs to active learning biascited by 8
  18. 2023Scaling Vision-based End-to-End Driving with Multi-View Attention Learningcited by 8
  19. 2024Active Fine-Tuning of Multi-Task Policiescited by 8
  20. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  21. 2022Markup-to-Image Diffusion Models with Scheduled Samplingcited by 6
  22. 2025Dynamic Rank Adjustment in Diffusion Policies for Efficient and Flexible Trainingcited by 6
  23. 2022Deconfounding Imitation Learning with Variational Inferencecited by 5
  24. 2025The Three Regimes of Offline-to-Online Reinforcement Learningcited by 4
  25. 2023Contrastive Example-Based Controlcited by 3
  26. 2025Diffusion-Based Approximate MPC: Fast and Consistent Imitation of Multi-Modal Action Distributionscited by 3
  27. 2023Theoretical Analysis of Offline Imitation With Supplementary Datasetcited by 2
  28. 2023Learning An Active Inference Model of Driver Perception and Control: Application to Vehicle Car-Followingcited by 2
  29. 2025A Simple Approach to Constraint-Aware Imitation Learning with Application to Autonomous Racingcited by 2
  30. 2021Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectoriescited by 1
  31. 2022CW-ERM: Improving Autonomous Driving Planning with Closed-loop Weighted Empirical Risk Minimizationcited by 1
  32. 2023GAN-MPC: Training Model Predictive Controllers with Parameterized Cost Functions using Demonstrations from Non-identical Expertscited by 1
  33. 2024Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloningcited by 1
  34. 2025Causal Imitation Learning under Expert-Observable and Expert-Unobservable Confoundingcited by 1
  35. 2025Synthetic vs. Real Training Data for Visual Navigationcited by 1
  36. 2025Toward Efficient and Robust Behavior Models for Multi-Agent Driving Simulationcited by 1
  37. 2026Beyond Self-Play: Hierarchical Reasoning for Continuous Motion in Closed-Loop Traffic Simulationcited by 1
  38. 2026Learning New Tasks via Reusable Skills: Skill-Compositional Experts for Embodied Continual Learningcited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Causal Transfer for Imitation Learning and Decision Making under Sensor-shift
  2. 2022Causal Imitation Learning with Unobserved Confounders
  3. 2026Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift
  4. 2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
  5. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances
  6. 2023Causal Imitability Under Context-Specific Independence Relations
  7. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
  8. 2025Action Chunking and Exploratory Data Collection Yield Exponential Improvements in Behavior Cloning for Continuous Control