# Imitation Learning by Reinforcement Learning

arXiv: https://arxiv.org/abs/2108.04763 · HTML: https://mlsanity.com/paper/2108.04763/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 10 Aug 2021.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 9 of 9)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation](https://mlsanity.com/paper/1905.06750/) (2019)

## Built on by (most cited first, 17 of 17)

- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [DITTO: Offline Imitation Learning with World Models](https://mlsanity.com/paper/2302.03086/) (2023)
- [SEABO: A Simple Search-Based Method for Offline Imitation Learning](https://mlsanity.com/paper/2402.03807/) (2024)
- [Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2402.08421/) (2024)
- [A Strong Baseline for Batch Imitation Learning](https://mlsanity.com/paper/2302.02788/) (2023)
- [When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training](https://mlsanity.com/paper/2509.24923/) (2025)
- [Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning](https://mlsanity.com/paper/2402.01057/) (2024)
- [Ask-AC: An Initiative Advisor-in-the-Loop Actor-Critic Framework](https://mlsanity.com/paper/2207.01955/) (2022)
- [Robot See, Robot Do: Imitation Reward for Noisy Financial Environments](https://mlsanity.com/paper/2411.08637/) (2024)
- [Reinforcement Learning for Hanabi](https://mlsanity.com/paper/2506.00458/) (2025)
- [Accelerated Continuous-Time Approximate Dynamic Programming via Data-Assisted Hybrid Control](https://mlsanity.com/paper/2204.12707/) (2022)
- [Backward Curriculum Reinforcement Learning](https://mlsanity.com/paper/2212.14214/) (2022)
- [Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments](https://mlsanity.com/paper/2310.05712/) (2023)
- [On the Complexity of Learning to Cooperate with Populations of Socially Rational Agents](https://mlsanity.com/paper/2407.00419/) (2024)
- [Imitation Learning of Correlated Policies in Stackelberg Games](https://mlsanity.com/paper/2503.08883/) (2025)
- [Social Cooperation in Conversational AI Agents](https://mlsanity.com/paper/2506.01624/) (2025)
- [Imitation Learning for Elder-Facing Speech Synthesis](https://mlsanity.com/paper/2606.21053/) (2026)

## Abstract (arXiv)

Imitation learning algorithms learn a policy from demonstrations of expert behavior. We show that, for deterministic experts, imitation learning can be done by reduction to reinforcement learning with a stationary reward. Our theoretical analysis both certifies the recovery of expert reward and bounds the total variation distance between the expert and the imitation learner, showing a link to adversarial imitation learning. We conduct experiments which confirm that our reduction works well in practice for continuous control tasks.