# Convergence of Value Aggregation for Imitation Learning

arXiv: https://arxiv.org/abs/1801.07292 · HTML: https://mlsanity.com/paper/1801.07292/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 6 of its 28 references that are in the map.
- First posted on arXiv 22 Jan 2018.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 6 of 6)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Reinforcement and Imitation Learning via Interactive No-Regret Learning](https://mlsanity.com/paper/1406.5979/) (2014)
- [Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction](https://mlsanity.com/paper/1703.01030/) (2017)
- [The Complexity of Large-scale Convex Programming under a Linear Optimization Oracle](https://mlsanity.com/paper/1309.5550/) (2013)
- [Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrations](https://mlsanity.com/paper/1610.00850/) (2016)
- [Agile Autonomous Driving using End-to-End Deep Imitation Learning](https://mlsanity.com/paper/1709.07174/) (2017)

## Built on by (most cited first, 7 of 7)

- [On-Policy Robot Imitation Learning from a Converging Supervisor](https://mlsanity.com/paper/1907.03423/) (2019)
- [Accelerating Imitation Learning with Predictive Models](https://mlsanity.com/paper/1806.04642/) (2018)
- [On Efficient Online Imitation Learning via Classification](https://mlsanity.com/paper/2209.12868/) (2022)
- [Dynamic Regret Convergence Analysis and an Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning](https://mlsanity.com/paper/1811.02184/) (2018)
- [Continuous Online Learning and New Insights to Online Imitation Learning](https://mlsanity.com/paper/1912.01261/) (2019)
- [Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning](https://mlsanity.com/paper/2412.07057/) (2024)
- [Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms](https://mlsanity.com/paper/2312.16860/) (2023)

## Abstract (arXiv)

Value aggregation is a general framework for solving imitation learning problems. Based on the idea of data aggregation, it generates a policy sequence by iteratively interleaving policy optimization and evaluation in an online learning setting. While the existence of a good policy in the policy sequence can be guaranteed non-asymptotically, little is known about the convergence of the sequence or the performance of the last policy. In this paper, we debunk the common belief that value aggregation always produces a convergent policy sequence with improving performance. Moreover, we identify a critical stability condition for convergence and provide a tight non-asymptotic bound on the performance of the last policy. These new theoretical insights let us stabilize problems with regularization, which removes the inconvenient process of identifying the best policy in the policy sequence in stochastic problems.