# Continuous Online Learning and New Insights to Online Imitation Learning

arXiv: https://arxiv.org/abs/1912.01261 · HTML: https://mlsanity.com/paper/1912.01261/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 3 Dec 2019.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 13 of 13)

- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [Non-stationary Stochastic Optimization](https://mlsanity.com/paper/1307.5449/) (2013)
- [Online Optimization : Competing with Dynamic Comparators](https://mlsanity.com/paper/1501.06225/) (2015)
- [Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction](https://mlsanity.com/paper/1703.01030/) (2017)
- [Tracking Slowly Moving Clairvoyant: Optimal Dynamic Regret of Online Learning with True and Noisy Gradient](https://mlsanity.com/paper/1605.04638/) (2016)
- [Improved Dynamic Regret for Non-degenerate Functions](https://mlsanity.com/paper/1608.03933/) (2016)
- [Online Learning with Inexact Proximal Online Gradient Descent Algorithms](https://mlsanity.com/paper/1806.00202/) (2018)
- [Online Learning with Continuous Variations: Dynamic Regret and Reductions](https://mlsanity.com/paper/1902.07286/) (2019)
- [A Reduction from Reinforcement Learning to No-Regret Online Learning](https://mlsanity.com/paper/1911.05873/) (2019)
- [Predictor-Corrector Policy Optimization](https://mlsanity.com/paper/1810.06509/) (2018)
- [Convergence of Value Aggregation for Imitation Learning](https://mlsanity.com/paper/1801.07292/) (2018)
- [Accelerating Imitation Learning with Predictive Models](https://mlsanity.com/paper/1806.04642/) (2018)
- [Dynamic Regret Convergence Analysis and an Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning](https://mlsanity.com/paper/1811.02184/) (2018)

## Built on by (most cited first, 2 of 2)

- [Learning from Imperfect Demonstrations from Agents with Varying Dynamics](https://mlsanity.com/paper/2103.05910/) (2021)
- [Explaining Fast Improvement in Online Imitation Learning](https://mlsanity.com/paper/2007.02520/) (2020)

## Abstract (arXiv)

Online learning is a powerful tool for analyzing iterative algorithms. However, the classic adversarial setup sometimes fails to capture certain regularity in online problems in practice. Motivated by this, we establish a new setup, called Continuous Online Learning (COL), where the gradient of online loss function changes continuously across rounds with respect to the learner's decisions. We show that COL covers and more appropriately describes many interesting applications, from general equilibrium problems (EPs) to optimization in episodic MDPs. Using this new setup, we revisit the difficulty of achieving sublinear dynamic regret. We prove that there is a fundamental equivalence between achieving sublinear dynamic regret in COL and solving certain EPs, and we present a reduction from dynamic regret to both static regret and convergence rate of the associated EP. At the end, we specialize these new insights into online imitation learning and show improved understanding of its learning stability.