Dynamic Regret Convergence Analysis and an Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
On-policy imitation learning algorithms such as DAgger evolve a robot control policy by executing it, measuring performance (loss), obtaining corrective feedback from a supervisor, and generating the next policy. As the loss between iterations can vary unpredictably, a fundamental question is under what conditions this process will eventually achieve a converged policy. If one assumes the underlying trajectory distribution is static (stationary), it is possible to prove convergence for DAgger. However, in more realistic models for robotics, the underlying trajectory distribution is dynamic because it is a function of the policy. Recent results show it is possible to prove convergence of DAgger when a regularity condition on the rate of change of the trajectory distributions is satisfied. In this article, we reframe this result using dynamic regret theory from the field of online optimization and show that dynamic regret can be applied to any on-policy algorithm to analyze its convergence and optimality. These results inspire a new algorithm, Adaptive On-Policy Regularization (AOR), that ensures the conditions for convergence. We present simulation results with cart-pole balancing and locomotion benchmarks that suggest AOR can significantly decrease dynamic regret and chattering as the robot learns. To our knowledge, this the first application of dynamic regret theory to imitation learning.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 8 of its 20 references that are in the map. First posted on arXiv 6 Nov 2018.
Builds on
8 of 20 references.
Closest ancestors · 3
Same direction, newest first
- 2018Accelerating Imitation Learning with Predictive Models12 Jun
- 2018Fast Policy Learning through Imitation and Reinforcement26 May
- 2018Convergence of Value Aggregation for Imitation Learning22 Jan
Other references · 5
- 2013Optimization, Learning, and Games with Predictable Sequencescited by 187
- 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
- 2016Tracking Slowly Moving Clairvoyant: Optimal Dynamic Regret of Online Learning with True and Noisy Gradientcited by 65
- 2016Improved Dynamic Regret for Non-degenerate Functionscited by 46
- 2016Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrationscited by 17
Built on by
Citing papers per year: 2019: 2; 2020: 0; 2021: 1.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2022Improved Policy Optimization for Online Imitation Learning
- 2024Contractive Dynamical Imitation Policies for Efficient Out-of-Sample Recovery
- 2025Dynamic Rank Adjustment in Diffusion Policies for Efficient and Flexible Training
- 2025The Pitfalls of Imitation Learning when Actions are Continuous
- 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport
- 2026Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret
- 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model
- 2025Elastic Motion Policy: An Adaptive Dynamical System for Robust and Efficient One-Shot Imitation Learning