Improved Policy Optimization for Online Imitation Learning

arXiv 2208.00088 ↗First posted 29 Jul 2022cs.LGcited by 6

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We consider online imitation learning (OIL), where the task is to find a policy that imitates the behavior of an expert via active interaction with the environment. We aim to bridge the gap between the theory and practice of policy optimization algorithms for OIL by analyzing one of the most popular OIL algorithms, DAGGER. Specifically, if the class of policies is sufficiently expressive to contain the expert policy, we prove that DAGGER achieves constant regret. Unlike previous bounds that require the losses to be strongly-convex, our result only requires the weaker assumption that the losses be strongly-convex with respect to the policy's sufficient statistics (not its parameterization). In order to ensure convergence for a wider class of policies and losses, we augment DAGGER with an additional regularization term. In particular, we propose a variant of Follow-the-Regularized-Leader (FTRL) and its adaptive variant for OIL and develop a memory-efficient implementation, which matches the memory requirements of FTL. Assuming that the loss functions are smooth and convex with respect to the parameters of the policy, we also prove that FTRL achieves constant regret for any sufficiently expressive policy class, while retaining O(√T) regret in the worst-case. We demonstrate the effectiveness of these algorithms with experiments on synthetic and high-dimensional control tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 79 references that are in the map. First posted on arXiv 29 Jul 2022.

Builds on

47 of 79 references.

Closest ancestors · 5

Same direction, newest first

2021Implicit Behavioral Cloning1 Sep
2021MobILE: Model-Based Imitation Learning From Observation Alone22 Feb
2020Explaining Fast Improvement in Online Imitation Learning6 Jul
2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
2019Provably Efficient Imitation Learning from Observation Alone27 May

Other references · 42

2014Adam: A Method for Stochastic Optimizationcited by 37,240
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2017Proximal Policy Optimization Algorithmscited by 15,215
2016Overcoming catastrophic forgetting in neural networkscited by 4,787
2015Continuous control with deep reinforcement learningcited by 3,407
2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2015Trust Region Policy Optimizationcited by 2,580
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
2019Introduction to Online Convex Optimizationcited by 803
2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
2018Learning Dexterous In-Hand Manipulationcited by 687

30 more

2017Generative Adversarial Networks: An Overviewcited by 590
2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
2019Learning by Cheatingcited by 383
2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
2017Inverse Reward Designcited by 249
2017DART: Noise Injection for Robust Imitation Learningcited by 191
2021Character Controllers Using Motion VAEscited by 184
2018Deep Reinforcement Learning and the Deadly Triadcited by 174
2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
2021Reward is enough for convex MDPscited by 84
2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
2018Online Adaptive Methods, Universality and Accelerationcited by 51
2018Feedback Control For Cassie With Deep Reinforcement Learningcited by 45
2017Data-driven Planning via Imitation Learningcited by 36
2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
2021Online Apprenticeship Learningcited by 29
2021Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Drivingcited by 27
2020Robust Asymmetric Learning in POMDPscited by 23
2021Scalable Online Planning via Reinforcement Learning Fine-Tuningcited by 18
2022Critic Sequential Monte Carlocited by 16
2019Apprenticeship Learning via Frank-Wolfecited by 13
2017On the linear convergence of the stochastic gradient method with constant step-sizecited by 6
2018Bandits with Side Observations: Bounded vs. Logarithmic Regretcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 2.

6 papers.

2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
2023Target-based Surrogates for Stochastic Optimizationcited by 4
2024MetaCURL: Non-stationary Concave Utility Reinforcement Learningcited by 4
2022Some Supervision Required: Incorporating Oracle Policies in Reinforcement Learning via Epistemic Uncertainty Metricscited by 0
2025Online Episodic Convex Reinforcement Learningcited by 0
2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0

Similar, not cited

Close in wording, no citation either way