# Fully General Online Imitation Learning

arXiv: https://arxiv.org/abs/2102.08686 · HTML: https://mlsanity.com/paper/2102.08686/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 5 of its 20 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 5 of 5)

- [EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning](https://mlsanity.com/paper/1807.08364/) (2018)
- [Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2007.12315/) (2020)
- [Risk-Aware Active Inverse Reinforcement Learning](https://mlsanity.com/paper/1901.02161/) (2019)
- [Pessimism About Unknown Unknowns Inspires Conservatism](https://mlsanity.com/paper/2006.08753/) (2020)
- [Asymptotically Unambitious Artificial General Intelligence](https://mlsanity.com/paper/1905.12186/) (2019)

## Built on by (most cited first, 4 of 4)

- [Can a Bayesian Oracle Prevent Harm from an Agent?](https://mlsanity.com/paper/2408.05284/) (2024)
- [RL, but don't do anything I wouldn't do](https://mlsanity.com/paper/2410.06213/) (2024)
- [Golden Handcuffs make safer AI agents](https://mlsanity.com/paper/2604.13609/) (2026)
- [YRC-Bench: A Benchmark for Learning to Coordinate with Experts](https://mlsanity.com/paper/2502.09583/) (2025)

## Abstract (arXiv)

In imitation learning, imitators and demonstrators are policies for picking actions given past interactions with the environment. If we run an imitator, we probably want events to unfold similarly to the way they would have if the demonstrator had been acting the whole time. In general, one mistake during learning can lead to completely different events. In the special setting of environments that restart, existing work provides formal guidance in how to imitate so that events unfold similarly, but outside that setting, no formal guidance exists. We address a fully general setting, in which the (stochastic) environment and demonstrator never reset, not even for training purposes, and we allow our imitator to learn online from the demonstrator. Our new conservative Bayesian imitation learner underestimates the probabilities of each available action, and queries for more data with the remaining probability. Our main result: if an event would have been unlikely had the demonstrator acted the whole time, that event's likelihood can be bounded above when running the (initially totally ignorant) imitator instead. Meanwhile, queries to the demonstrator rapidly diminish in frequency. If any such event qualifies as "dangerous", our imitator would have the notable distinction of being relatively "safe".