# Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization

arXiv: https://arxiv.org/abs/1809.00403 · HTML: https://mlsanity.com/paper/1809.00403/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Sep 2018.

## Built on by (most cited first, 8 of 8)

- [A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals](https://mlsanity.com/paper/2408.05804/) (2024)
- [Ensemble Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.02618/) (2023)
- [A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search](https://mlsanity.com/paper/2202.07071/) (2022)
- [Ensemble Reinforcement Learning in Continuous Spaces -- A Hierarchical Multi-Step Approach for Policy Training](https://mlsanity.com/paper/2209.14488/) (2022)
- [Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence](https://mlsanity.com/paper/2301.11476/) (2023)
- [q-Munchausen Reinforcement Learning](https://mlsanity.com/paper/2205.07467/) (2022)
- [Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning](https://mlsanity.com/paper/2205.07885/) (2022)
- [Tsallis Entropy Regularization for Linear Quadratic Regulator and Kullback-Leibler Control](https://mlsanity.com/paper/2403.01805/) (2024)

## Abstract (arXiv)

Recently deep reinforcement learning (DRL) has achieved outstanding success on solving many difficult and large-scale RL problems. However the high sample cost required for effective learning often makes DRL unaffordable in resource-limited applications. With the aim of improving sample efficiency and learning performance, we will develop a new DRL algorithm in this paper that seamless integrates entropy-induced and bootstrap-induced techniques for efficient and deep exploration of the learning environment. Specifically, a general form of Tsallis entropy regularizer will be utilized to drive entropy-induced exploration based on efficient approximation of optimal action-selection policies. Different from many existing works that rely on action dithering strategies for exploration, our algorithm is efficient in exploring actions with clear exploration value. Meanwhile, by employing an ensemble of Q-networks under varied Tsallis entropy regularization, the diversity of the ensemble can be further enhanced to enable effective bootstrap-induced exploration. Experiments on Atari game playing tasks clearly demonstrate that our new algorithm can achieve more efficient and effective exploration for DRL, in comparison to recently proposed exploration methods including Bootstrapped Deep Q-Network and UCB Q-Ensemble.