# Fast Policy Learning through Imitation and Reinforcement

arXiv: https://arxiv.org/abs/1805.10413 · HTML: https://mlsanity.com/paper/1805.10413/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 26 May 2018.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 27 of 27)

- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning](https://mlsanity.com/paper/1910.12179/) (2019)
- [Learning to Generate Better Than Your LLM](https://mlsanity.com/paper/2306.11816/) (2023)
- [RLIF: Interactive Imitation Learning as Reinforcement Learning](https://mlsanity.com/paper/2311.12996/) (2023)
- [SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning](https://mlsanity.com/paper/2403.09110/) (2024)
- [Blending Imitation and Reinforcement Learning for Robust Policy Improvement](https://mlsanity.com/paper/2310.01737/) (2023)
- [Robots Enact Malignant Stereotypes](https://mlsanity.com/paper/2207.11569/) (2022)
- [How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2205.03353/) (2022)
- [Combining imitation and deep reinforcement learning to accomplish human-level performance on a virtual foraging task](https://mlsanity.com/paper/2203.06250/) (2022)
- [Self-Imitation Learning from Demonstrations](https://mlsanity.com/paper/2203.10905/) (2022)
- [Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving](https://mlsanity.com/paper/2603.13842/) (2026)
- [Dynamic Regret Convergence Analysis and an Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning](https://mlsanity.com/paper/1811.02184/) (2018)
- [Bootstrapping Motor Skill Learning with Motion Planning](https://mlsanity.com/paper/2101.04736/) (2021)
- [Online Baum-Welch algorithm for Hierarchical Imitation Learning](https://mlsanity.com/paper/2103.12197/) (2021)
- [Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making](https://mlsanity.com/paper/2512.08280/) (2025)
- [Deploying Offline Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2303.07046/) (2023)
- [Explaining Fast Improvement in Online Imitation Learning](https://mlsanity.com/paper/2007.02520/) (2020)
- [Generalizing to New Tasks via One-Shot Compositional Subgoals](https://mlsanity.com/paper/2205.07716/) (2022)
- [Transferring Multiple Policies to Hotstart Reinforcement Learning in an Air Compressor Management Problem](https://mlsanity.com/paper/2301.12820/) (2023)
- [A Structured Prediction Approach for Robot Imitation Learning](https://mlsanity.com/paper/2309.14829/) (2023)
- [Learning Neural Parsers with Deterministic Differentiable Imitation Learning](https://mlsanity.com/paper/1806.07822/) (2018)
- [Leveraging Experience in Lazy Search](https://mlsanity.com/paper/2110.04669/) (2021)
- [Neural-Progressive Hedging: Enforcing Constraints in Reinforcement Learning with Stochastic Programming](https://mlsanity.com/paper/2202.13436/) (2022)
- [Accelerating Inverse Reinforcement Learning with Expert Bootstrapping](https://mlsanity.com/paper/2402.02608/) (2024)
- [A Model-Based Approach for Improving Reinforcement Learning Efficiency Leveraging Expert Observations](https://mlsanity.com/paper/2402.18836/) (2024)
- [TubeDAgger: Reducing the Number of Expert Interventions with Stochastic Reach-Tubes](https://mlsanity.com/paper/2510.00906/) (2025)
- [Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning](https://mlsanity.com/paper/2610.06019/) (2026)

## Abstract (arXiv)

Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.