# Scalable Multi-Task Imitation Learning with Autonomous Improvement

arXiv: https://arxiv.org/abs/2003.02636 · HTML: https://mlsanity.com/paper/2003.02636/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 25 Feb 2020.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning](https://mlsanity.com/paper/1802.01557/) (2018)
- [End-to-End Robotic Reinforcement Learning without Reward Engineering](https://mlsanity.com/paper/1904.07854/) (2019)
- [Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration](https://mlsanity.com/paper/1707.02920/) (2017)
- [Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction](https://mlsanity.com/paper/1703.01030/) (2017)
- [Task-Embedded Control Networks for Few-Shot Imitation Learning](https://mlsanity.com/paper/1810.03237/) (2018)
- [Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration](https://mlsanity.com/paper/1807.03480/) (2018)
- [Hindsight policy gradients](https://mlsanity.com/paper/1711.06006/) (2017)
- [Watch, Try, Learn: Meta-Learning from Demonstrations and Reward](https://mlsanity.com/paper/1906.03352/) (2019)
- [One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL](https://mlsanity.com/paper/1810.05017/) (2018)
- [Unsupervised Visuomotor Control through Distributional Planning Networks](https://mlsanity.com/paper/1902.05542/) (2019)

## Built on by (most cited first, 21 of 21)

- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos](https://mlsanity.com/paper/2103.16817/) (2021)
- [SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution](https://mlsanity.com/paper/2312.11598/) (2023)
- [Imitation Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2202.01288/) (2022)
- [Towards More Generalizable One-shot Visual Imitation Learning](https://mlsanity.com/paper/2110.13423/) (2021)
- [The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models](https://mlsanity.com/paper/2204.10759/) (2022)
- [Multi-Task Imitation Learning for Linear Dynamical Systems](https://mlsanity.com/paper/2212.00186/) (2022)
- [SOP: A Scalable Online Post-Training System for Vision-Language-Action Models](https://mlsanity.com/paper/2601.03044/) (2026)
- [Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2210.01969/) (2022)
- [Multi-task Hierarchical Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/2305.12633/) (2023)
- [Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data](https://mlsanity.com/paper/2308.04428/) (2023)
- [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023)
- [Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards](https://mlsanity.com/paper/2010.06962/) (2020)
- [Optimal Actor-Critic Policy with Optimized Training Datasets](https://mlsanity.com/paper/2108.06911/) (2021)
- [Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Division](https://mlsanity.com/paper/2203.14855/) (2022)
- [Abstract-to-Executable Trajectory Translation for One-Shot Task Generalization](https://mlsanity.com/paper/2210.07658/) (2022)
- [Offline Imitation of Badminton Player Behavior via Experiential Contexts and Brownian Motion](https://mlsanity.com/paper/2403.12406/) (2024)
- [Data-Efficient Multitask DAgger](https://mlsanity.com/paper/2509.25466/) (2025)
- [Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learning](https://mlsanity.com/paper/2406.00761/) (2024)

## Abstract (arXiv)

While robot learning has demonstrated promising results for enabling robots to automatically acquire new skills, a critical challenge in deploying learning-based systems is scale: acquiring enough data for the robot to effectively generalize broadly. Imitation learning, in particular, has remained a stable and powerful approach for robot learning, but critically relies on expert operators for data collection. In this work, we target this challenge, aiming to build an imitation learning system that can continuously improve through autonomous data collection, while simultaneously avoiding the explicit use of reinforcement learning, to maintain the stability, simplicity, and scalability of supervised imitation. To accomplish this, we cast the problem of imitation with autonomous improvement into a multi-task setting. We utilize the insight that, in a multi-task setting, a failed attempt at one task might represent a successful attempt at another task. This allows us to leverage the robot's own trials as demonstrations for tasks other than the one that the robot actually attempted. Using an initial dataset of multi-task demonstration data, the robot autonomously collects trials which are only sparsely labeled with a binary indication of whether the trial accomplished any useful task or not. We then embed the trials into a learned latent space of tasks, trained using only the initial demonstration dataset, to draw similarities between various trials, enabling the robot to achieve one-shot generalization to new tasks. In contrast to prior imitation learning approaches, our method can autonomously collect data with sparse supervision for continuous improvement, and in contrast to reinforcement learning algorithms, our method can effectively improve from sparse, task-agnostic reward signals.