mlsanityData to 7 Oct 2026

On-Policy Robot Imitation Learning from a Converging Supervisor

arXiv 1907.03423 ↗First posted 8 Jul 2019cs.LG cs.AI cs.ROcited by 10

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Existing on-policy imitation learning algorithms, such as DAgger, assume access to a fixed supervisor. However, there are many settings where the supervisor may evolve during policy learning, such as a human performing a novel task or an improving algorithmic controller. We formalize imitation learning from a "converging supervisor" and provide sublinear static and dynamic regret guarantees against the best policy in hindsight with labels from the converged supervisor, even when labels during learning are only from intermediate supervisors. We then show that this framework is closely connected to a class of reinforcement learning (RL) algorithms known as dual policy iteration (DPI), which alternate between training a reactive learner with imitation learning and a model-based supervisor with data from the learner. Experiments suggest that when this framework is applied with the state-of-the-art deep model-based RL algorithm PETS as an improving supervisor, it outperforms deep RL baselines on continuous control tasks and provides up to an 80-fold speedup in policy evaluation.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 35 references that are in the map. First posted on arXiv 8 Jul 2019.

Builds on

22 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Dynamic Regret Convergence Analysis and an Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning6 Nov
  2. 2018Convergence of Value Aggregation for Imitation Learning22 Jan

Other references · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  7. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  9. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  10. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  11. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  12. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
8 more
  1. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  2. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  3. 2016PLATO: Policy Learning using Adaptive Trajectory Optimizationcited by 23
  4. 2018Dual Policy Iterationcited by 22
  5. 2016Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrationscited by 17
  6. 2017Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedurecited by 11
  7. 2019Online Learning with Continuous Variations: Dynamic Regret and Reductionscited by 10
  8. 2019Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Taskscited by 3

Built on by

Cited by year

Citing papers per year: 2020: 1; 2021: 6; 2022: 1; 2023: 0; 2024: 1; 2025: 1.

10 papers.

  1. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  2. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  3. 2021VisuoSpatial Foresight for Physical Sequential Fabric Manipulationcited by 18
  4. 2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
  5. 2021Learning from an Exploring Demonstrator: Optimal Reward Estimation for Banditscited by 8
  6. 2020Inverse Reinforcement Learning from a Gradient-based Learnercited by 7
  7. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 5
  8. 2021Behavior Self-Organization Supports Task Inference for Continual Robot Learningcited by 4
  9. 2022Learning Solution Manifolds for Control Problems via Energy Minimizationcited by 1
  10. 2024IBCB: Efficient Inverse Batched Contextual Bandit for Behavioral Evolution Historycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model
  2. 2018Fast Policy Learning through Imitation and Reinforcement
  3. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policies
  4. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
  5. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
  6. 2025IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning
  7. 2019Reward-Conditioned Policies
  8. 2021Implicit Behavioral Cloning