mlsanityData to 7 Oct 2026

Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning

arXiv 2412.07057 ↗First posted 9 Dec 2024stat.ML cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Imitation learning (IL) is a paradigm for learning sequential decision making policies from experts, leveraging offline demonstrations, interactive annotations, or both. Recent advances show that when annotation cost is tallied per trajectory, Behavior Cloning (BC) which relies solely on offline demonstrations cannot be improved in general, leaving limited conditions for interactive methods such as DAgger to help. We revisit this conclusion and prove that when the annotation cost is measured per state, algorithms using interactive annotations can provably outperform BC. Specifically: (1) we show that Stagger, a one sample per round variant of DAgger, provably beats BC under low recovery cost settings; (2) we initiate the study of hybrid IL where the agent learns from offline demonstrations and interactive annotations. We propose Warm Stagger whose learning guarantee is not much worse than using either data source alone. Furthermore, motivated by compounding error and cold start problem in imitation learning practice, we give an MDP example in which Warm Stagger has significant better annotation cost; (3) experiments on MuJoCo continuous control tasks confirm that, with modest cost ratio between interactive and offline annotations, interactive and hybrid approaches consistently outperform BC. To the best of our knowledge, our work is the first to highlight the benefit of state wise interactive annotation and hybrid feedback in imitation learning.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 84 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

50 of 84 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning20 Jul
  2. 2023Selective Sampling and Imitation Learning via Online Regression11 Jul
  3. 2022On Efficient Online Imitation Learning via Classification26 Sep
  4. 2022Minimax Optimal Online Imitation Learning via Replay Estimation30 May
  5. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning17 Sep
  6. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift4 Feb
  7. 2020Toward the Fundamental Limits of Imitation Learning13 Sep
  8. 2020Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms14 Jun
  9. 2019Imitation Learning as f-Divergence Minimization30 May
  10. 2019Provably Efficient Imitation Learning from Observation Alone27 May
  11. 2019Uncertainty-Aware Data Aggregation for Deep Imitation Learning7 May
  12. 2018An Algorithmic Perspective on Imitation Learning16 Nov
5 more
  1. 2018Predictor-Corrector Policy Optimization15 Oct
  2. 2018HG-DAgger: Interactive Imitation Learning with Human Experts5 Oct
  3. 2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning22 Jul
  4. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning29 May
  5. 2018Convergence of Value Aggregation for Imitation Learning22 Jan

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2016End to End Learning for Self-Driving Carscited by 1,000
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  9. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  10. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  11. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  12. 2019Self-Driving Cars: A Surveycited by 147
21 more
  1. 2013Online Learning under Delayed Feedbackcited by 137
  2. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  3. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  4. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  5. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  6. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  7. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  8. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  9. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  10. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  11. 2020Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulationcited by 37
  12. 2024Hybrid Inverse Reinforcement Learningcited by 31
  13. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  14. 2019Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedbackcited by 21
  15. 2016Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrationscited by 17
  16. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learningcited by 15
  17. 2017DropoutDAgger: A Bayesian Approach to Safe Imitation Learningcited by 14
  18. 2021Online Strongly Convex Optimization with Unknown Delayscited by 11
  19. 2022A Few Expert Queries Suffices for Sample-Efficient RL with Resets and Linear Value Approximationcited by 7
  20. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPscited by 6
  21. 2023Real-time Learning of Driving Gap Preference for Personalized Adaptive Cruise Controlcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DISEIL: Demonstration Distillation for Sample-Efficient Imitation Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms
  2. 2026Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift
  3. 2020Imitating Unknown Policies via Exploration
  4. 2026Difference-Aware Retrieval Policies for Imitation Learning
  5. 2025A Model-Based Approach to Imitation Learning through Multi-Step Predictions
  6. 2023Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior
  7. 2023A Strong Baseline for Batch Imitation Learning
  8. 2025Action Chunking and Exploratory Data Collection Yield Exponential Improvements in Behavior Cloning for Continuous Control