# Feedback in Imitation Learning: The Three Regimes of Covariate Shift

arXiv: https://arxiv.org/abs/2102.02872 · HTML: https://mlsanity.com/paper/2102.02872/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 4 Feb 2021.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 11 of 11)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Fighting Copycat Agents in Behavioral Cloning from Observation Histories](https://mlsanity.com/paper/2010.14876/) (2020)
- [Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization](https://mlsanity.com/paper/2006.13258/) (2020)
- [Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrations](https://mlsanity.com/paper/1610.00850/) (2016)

## Built on by (most cited first, 30 of 68)

- [Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap](https://mlsanity.com/paper/2103.03236/) (2021)
- [Data Quality in Imitation Learning](https://mlsanity.com/paper/2306.02437/) (2023)
- [HYDRA: Hybrid Robot Actions for Imitation Learning](https://mlsanity.com/paper/2306.17237/) (2023)
- [Visual Adversarial Imitation Learning using Variational Models](https://mlsanity.com/paper/2107.08829/) (2021)
- [Causal Reinforcement Learning using Observational and Interventional Data](https://mlsanity.com/paper/2106.14421/) (2021)
- [ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context](https://mlsanity.com/paper/2510.04246/) (2025)
- [Sequence Model Imitation Learning with Unobserved Contexts](https://mlsanity.com/paper/2208.02225/) (2022)
- [LLF-Bench: Benchmark for Interactive Learning from Language Feedback](https://mlsanity.com/paper/2312.06853/) (2023)
- [Learning Long-Context Diffusion Policies via Past-Token Prediction](https://mlsanity.com/paper/2505.09561/) (2025)
- [CCIL: Continuity-based Data Augmentation for Corrective Imitation Learning](https://mlsanity.com/paper/2310.12972/) (2023)
- [Causal Imitation Learning under Temporally Correlated Noise](https://mlsanity.com/paper/2202.01312/) (2022)
- [Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage](https://mlsanity.com/paper/2106.03207/) (2021)
- [RoboCopilot: Human-in-the-loop Interactive Imitation Learning for Robot Manipulation](https://mlsanity.com/paper/2503.07771/) (2025)
- [Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching](https://mlsanity.com/paper/2303.02569/) (2023)
- [Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models](https://mlsanity.com/paper/2409.16663/) (2024)
- [A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem](https://mlsanity.com/paper/2305.17198/) (2023)
- [Discriminator-Guided Model-Based Offline Imitation Learning](https://mlsanity.com/paper/2207.00244/) (2022)
- [Rethinking ValueDice: Does It Really Improve Performance?](https://mlsanity.com/paper/2202.02468/) (2022)
- [Fighting Fire with Fire: Avoiding DNN Shortcuts through Priming](https://mlsanity.com/paper/2206.10816/) (2022)
- [Resolving Copycat Problems in Visual Imitation Learning via Residual Action Prediction](https://mlsanity.com/paper/2207.09705/) (2022)
- [FutureDepth: Learning to Predict the Future Improves Video Depth Estimation](https://mlsanity.com/paper/2403.12953/) (2024)
- [Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecification](https://mlsanity.com/paper/2502.12465/) (2025)
- [Visual Backtracking Teleoperation: A Data Collection Protocol for Offline Image-Based Reinforcement Learning](https://mlsanity.com/paper/2210.02343/) (2022)
- [Efficient Imitation Learning with Conservative World Models](https://mlsanity.com/paper/2405.13193/) (2024)
- [Multi-Agent Imitation Learning: Value is Easy, Regret is Hard](https://mlsanity.com/paper/2406.04219/) (2024)
- [Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedback](https://mlsanity.com/paper/2410.08852/) (2024)
- [Receding Horizon Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.04477/) (2022)
- [A Bayesian Approach to Robust Inverse Reinforcement Learning](https://mlsanity.com/paper/2309.08571/) (2023)
- [Active Fine-Tuning of Multi-Task Policies](https://mlsanity.com/paper/2410.05026/) (2024)
- [Characterizing the robustness of Bayesian adaptive experimental designs to active learning bias](https://mlsanity.com/paper/2205.13698/) (2022)

## Abstract (arXiv)

Imitation learning practitioners have often noted that conditioning policies on previous actions leads to a dramatic divergence between "held out" error and performance of the learner in situ. Interactive approaches can provably address this divergence but require repeated querying of a demonstrator. Recent work identifies this divergence as stemming from a "causal confound" in predicting the current action, and seek to ablate causal aspects of current state using tools from causal inference. In this work, we argue instead that this divergence is simply another manifestation of covariate shift, exacerbated particularly by settings of feedback between decisions and input features. The learner often comes to rely on features that are strongly predictive of decisions, but are subject to strong covariate shift. Our work demonstrates a broad class of problems where this shift can be mitigated, both theoretically and practically, by taking advantage of a simulator but without any further querying of expert demonstration. We analyze existing benchmarks used to test imitation learning approaches and find that these benchmarks are realizable and simple and thus insufficient for capturing the harder regimes of error compounding seen in real-world decision making problems. We find, in a surprising contrast with previous literature, but consistent with our theory, that naive behavioral cloning provides excellent results. We detail the need for new standardized benchmarks that capture the phenomena seen in robotics problems.