# Ensemble Reinforcement Learning: A Survey

arXiv: https://arxiv.org/abs/2303.02618 · HTML: https://mlsanity.com/paper/2303.02618/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 124 references that are in the map.
- First posted on arXiv 5 Mar 2023.

## Builds on (most cited first, 30 of 43)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion](https://mlsanity.com/paper/1807.01675/) (2018)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning](https://mlsanity.com/paper/2102.13651/) (2021)
- [Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review](https://mlsanity.com/paper/2102.04148/) (2021)
- [Ensemble Bootstrapping for Q-Learning](https://mlsanity.com/paper/2103.00445/) (2021)
- [Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives](https://mlsanity.com/paper/1906.10667/) (2019)
- [Random Vector Functional Link Neural Network based Ensemble Deep Learning](https://mlsanity.com/paper/1907.00350/) (2019)
- [Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation](https://mlsanity.com/paper/2107.01825/) (2021)
- [Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble](https://mlsanity.com/paper/2205.09284/) (2022)
- [Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-time Popularities](https://mlsanity.com/paper/1708.06698/) (2017)
- [SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning](https://mlsanity.com/paper/2102.11075/) (2021)
- [Ensemble-Based Deep Reinforcement Learning for Chatbots](https://mlsanity.com/paper/1908.10422/) (2019)
- [Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games](https://mlsanity.com/paper/1909.08735/) (2019)
- [SEERL: Sample Efficient Ensemble Reinforcement Learning](https://mlsanity.com/paper/2001.05209/) (2020)
- [ElegantRL-Podracer: Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning](https://mlsanity.com/paper/2112.05923/) (2021)
- [Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization](https://mlsanity.com/paper/1809.00403/) (2018)

## Built on by (most cited first, 16 of 16)

- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities](https://mlsanity.com/paper/2308.13420/) (2023)
- [Graph-enabled Reinforcement Learning for Time Series Forecasting with Adaptive Intelligence](https://mlsanity.com/paper/2309.10186/) (2023)
- [Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions](https://mlsanity.com/paper/2410.11833/) (2024)
- [Fusion-PSRO: Nash Policy Fusion for Policy Space Response Oracles](https://mlsanity.com/paper/2405.21027/) (2024)
- [Progressive-Resolution Policy Distillation: Leveraging Coarse-Resolution Simulations for Time-Efficient Fine-Resolution Policy Learning](https://mlsanity.com/paper/2412.07477/) (2024)
- [Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents](https://mlsanity.com/paper/2502.04492/) (2025)
- [Dataset Clustering for Improved Offline Policy Learning](https://mlsanity.com/paper/2402.09550/) (2024)
- [vMFER: Von Mises-Fisher Experience Resampling Based on Uncertainty of Gradient Directions for Policy Improvement](https://mlsanity.com/paper/2405.08638/) (2024)
- [Oracle-Efficient Reinforcement Learning for Max Value Ensembles](https://mlsanity.com/paper/2405.16739/) (2024)
- [Prediction Instability in Machine Learning Ensembles](https://mlsanity.com/paper/2407.03194/) (2024)
- [Multi-Objective Reinforcement Learning for Large Language Model Optimization: Visionary Perspective](https://mlsanity.com/paper/2509.21613/) (2025)
- [Partial Models for Building Adaptive Model-Based Reinforcement Learning Agents](https://mlsanity.com/paper/2405.16899/) (2024)
- [Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme One](https://mlsanity.com/paper/2505.15306/) (2025)
- [Decision Flow Policy Optimization](https://mlsanity.com/paper/2505.20350/) (2025)
- [Quantum-Inspired Reinforcement Learning for Secure and Sustainable AIoT-Driven Supply Chain Systems](https://mlsanity.com/paper/2601.22339/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) has emerged as a highly effective technique for addressing various scientific and applied problems. Despite its success, certain complex tasks remain challenging to be addressed solely with a single model and algorithm. In response, ensemble reinforcement learning (ERL), a promising approach that combines the benefits of both RL and ensemble learning (EL), has gained widespread popularity. ERL leverages multiple models or training algorithms to comprehensively explore the problem space and possesses strong generalization capabilities. In this study, we present a comprehensive survey on ERL to provide readers with an overview of recent advances and challenges in the field. Firstly, we provide an introduction to the background and motivation for ERL. Secondly, we conduct a detailed analysis of strategies such as model selection and combination that have been successfully implemented in ERL. Subsequently, we explore the application of ERL, summarize the datasets, and analyze the algorithms employed. Finally, we outline several open questions and discuss future research directions of ERL. By offering guidance for future scientific research and engineering applications, this survey significantly contributes to the advancement of ERL.