# ε-BMC: A Bayesian Ensemble Approach to Epsilon-Greedy Exploration in Model-Free Reinforcement Learning

arXiv: https://arxiv.org/abs/2007.00869 · HTML: https://mlsanity.com/paper/2007.00869/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2020.

## Built on by (most cited first, 4 of 4)

- [LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Framework](https://mlsanity.com/paper/2310.03342/) (2023)
- [A step toward a reinforcement learning de novo genome assembler](https://mlsanity.com/paper/2102.02649/) (2021)
- [Dealing with uncertainty: balancing exploration and exploitation in deep recurrent reinforcement learning](https://mlsanity.com/paper/2310.08331/) (2023)
- [Sample-Efficient Neurosymbolic Deep Reinforcement Learning](https://mlsanity.com/paper/2601.02850/) (2026)

## Abstract (arXiv)

Resolving the exploration-exploitation trade-off remains a fundamental problem in the design and implementation of reinforcement learning (RL) algorithms. In this paper, we focus on model-free RL using the epsilon-greedy exploration policy, which despite its simplicity, remains one of the most frequently used forms of exploration. However, a key limitation of this policy is the specification of varepsilon. In this paper, we provide a novel Bayesian perspective of varepsilon as a measure of the uniformity of the Q-value function. We introduce a closed-form Bayesian model update based on Bayesian model combination (BMC), based on this new perspective, which allows us to adapt varepsilon using experiences from the environment in constant time with monotone convergence guarantees. We demonstrate that our proposed algorithm, varepsilon-textttBMC, efficiently balances exploration and exploitation on different problems, performing comparably or outperforming the best tuned fixed annealing schedules and an alternative data-dependent varepsilon adaptation scheme proposed in the literature.