# Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

arXiv: https://arxiv.org/abs/1903.04193 · HTML: https://mlsanity.com/paper/1903.04193/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2019.

## Built on by (most cited first, 3 of 3)

- [SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2308.02594/) (2023)
- [Synthesising Reinforcement Learning Policies through Set-Valued Inductive Rule Learning](https://mlsanity.com/paper/2106.06009/) (2021)
- [Dynamic Weight Adjusting Deep Q-Networks for Real-Time Environmental Adaptation](https://mlsanity.com/paper/2411.02559/) (2024)

## Abstract (arXiv)

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.