mlsanityData to 7 Oct 2026

SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning

arXiv 2007.04938 ↗First posted 9 Jul 2020cs.LG cs.AI stat.MLcited by 150

Not on the map. Why?

Cited by 25 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Off-policy deep reinforcement learning (RL) has been successful in a range of challenging domains. However, standard off-policy RL algorithms can suffer from several issues, such as instability in Q-learning and balancing exploration and exploitation. To mitigate these issues, we present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy RL algorithms. SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration. By enforcing the diversity between agents using Bootstrap with random initialization, we show that these different ideas are largely orthogonal and can be fruitfully integrated, together further improving the performance of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, for both continuous and discrete control tasks on both low-dimensional and high-dimensional environments. Our training code is available at https://github.com/pokaxpoka/sunrise.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 31; 2023: 34; 2024: 24; 2025: 19; 2026: 15.

150 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  5. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  6. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  7. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  8. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  9. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  10. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  11. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  12. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
38 more
  1. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  2. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  3. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  4. 2023Thompson sampling for improved exploration in GFlowNetscited by 31
  5. 2021Ensemble Bootstrapping for Q-Learningcited by 30
  6. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  7. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  8. 2022Sample Efficient Deep Reinforcement Learning via Uncertainty Estimationcited by 26
  9. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  10. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 21
  11. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  12. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  13. 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
  14. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  15. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  16. 2022Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemblecited by 18
  17. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  18. 2021Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitationcited by 17
  19. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  20. 2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RLcited by 17
  21. 2022Critic Sequential Monte Carlocited by 16
  22. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  23. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12
  24. 2022Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixelscited by 12
  25. 2023A Survey of Demonstration Learningcited by 12
  26. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  27. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  28. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  29. 2023Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agentscited by 12
  30. 2025Value Flowscited by 12
  31. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
  32. 2021False Correlation Reduction for Offline Reinforcement Learningcited by 9
  33. 2023Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedbackcited by 9
  34. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 9
  35. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  36. 2021Continuous Control With Ensemble Deep Deterministic Policy Gradientscited by 8
  37. 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8
  38. 2022The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Realitycited by 8

50 most cited of 150 All 150 (JSON)