# Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations

arXiv: https://arxiv.org/abs/1801.10459 · HTML: https://mlsanity.com/paper/1801.10459/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2018.

## Built on by (most cited first, 7 of 7)

- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Imitation Learning with Concurrent Actions in 3D Games](https://mlsanity.com/paper/1803.05402/) (2018)
- [Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations](https://mlsanity.com/paper/2305.13030/) (2023)
- [Deep Reinforcement Learning in Autonomous Car Path Planning and Control: A Survey](https://mlsanity.com/paper/2404.00340/) (2024)
- [Cut-and-Approximate: 3D Shape Reconstruction from Planar Cross-sections with Deep Reinforcement Learning](https://mlsanity.com/paper/2210.12509/) (2022)
- [Some Supervision Required: Incorporating Oracle Policies in Reinforcement Learning via Epistemic Uncertainty Metrics](https://mlsanity.com/paper/2208.10533/) (2022)
- [Enhancing team performance with transfer-learning during real-world human-robot collaboration](https://mlsanity.com/paper/2211.13070/) (2022)

## Abstract (arXiv)

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert demonstrations. However, these methods are unstable and not suitable for actor-critic reinforcement learning algorithms. Also, some existing methods rely on the global optimum assumption, which is not true in most scenarios. In this paper, we employ expert demonstrations in a actor-critic reinforcement learning framework, and meanwhile ensure that the performance is not affected by the fact that expert demonstrations are not global optimal. We theoretically derive a method for computing policy gradients and value estimators with only expert demonstrations. Our method is theoretically plausible for actor-critic reinforcement learning algorithms that pretrains both policy and value functions. We apply our method to two of the typical actor-critic reinforcement learning algorithms, DDPG and ACER, and demonstrate with experiments that our method not only outperforms the RL algorithms without pretraining process, but also is more simulation efficient.