# Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments

arXiv: https://arxiv.org/abs/1910.04281 · HTML: https://mlsanity.com/paper/1910.04281/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 17 of 17)

- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot Learning](https://mlsanity.com/paper/2301.00452/) (2023)
- [A Survey on Reinforcement Learning Methods in Character Animation](https://mlsanity.com/paper/2203.04735/) (2022)
- [Behavior Prior Representation learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2211.00863/) (2022)
- [Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation](https://mlsanity.com/paper/2111.06383/) (2021)
- [Disturbance-Injected Robust Imitation Learning with Task Achievement](https://mlsanity.com/paper/2205.04195/) (2022)
- [Evolutionary Action Selection for Gradient-based Policy Learning](https://mlsanity.com/paper/2201.04286/) (2022)
- [Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control](https://mlsanity.com/paper/2210.01525/) (2022)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets](https://mlsanity.com/paper/2212.02125/) (2022)
- [Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft](https://mlsanity.com/paper/2112.03482/) (2021)
- [Sample-Efficient Multi-Agent Reinforcement Learning with Demonstrations for Flocking Control](https://mlsanity.com/paper/2209.08351/) (2022)
- [Specialized Deep Residual Policy Safe Reinforcement Learning-Based Controller for Complex and Continuous State-Action Spaces](https://mlsanity.com/paper/2310.14788/) (2023)
- [Data-Driven Simulation of Ride-Hailing Services using Imitation and Reinforcement Learning](https://mlsanity.com/paper/2104.02661/) (2021)
- [Improving Learning from Demonstrations by Learning from Experience](https://mlsanity.com/paper/2111.08156/) (2021)
- [Sub-optimal Policy Aided Multi-Agent Reinforcement Learning for Flocking Control](https://mlsanity.com/paper/2209.08347/) (2022)
- [CueLearner: Bootstrapping and local policy adaptation from relative feedback](https://mlsanity.com/paper/2507.04730/) (2025)

## Abstract (arXiv)

This paper investigates how to efficiently transition and update policies, trained initially with demonstrations, using off-policy actor-critic reinforcement learning. It is well-known that techniques based on Learning from Demonstrations, for example behavior cloning, can lead to proficient policies given limited data. However, it is currently unclear how to efficiently update that policy using reinforcement learning as these approaches are inherently optimizing different objective functions. Previous works have used loss functions, which combine behavior cloning losses with reinforcement learning losses to enable this update. However, the components of these loss functions are often set anecdotally, and their individual contributions are not well understood. In this work, we propose the Cycle-of-Learning (CoL) framework that uses an actor-critic architecture with a loss function that combines behavior cloning and 1-step Q-learning losses with an off-policy pre-training step from human demonstrations. This enables transition from behavior cloning to reinforcement learning without performance degradation and improves reinforcement learning in terms of overall performance and training time. Additionally, we carefully study the composition of these combined losses and their impact on overall policy learning. We show that our approach outperforms state-of-the-art techniques for combining behavior cloning and reinforcement learning for both dense and sparse reward scenarios. Our results also suggest that directly including the behavior cloning loss on demonstration data helps to ensure stable learning and ground future policy updates.