mlsanityData to 14 Oct 2026

Maxmin Q-learning: Controlling the Estimation Bias of Q-learning

arXiv 2002.06487 ↗First posted 16 Feb 2020cs.LG cs.AIcited by 117

Not on the map. Why?

Cited by 47 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Q-learning suffers from overestimation bias, because it approximates the maximum action value using the maximum estimated action value. Algorithms have been proposed to reduce overestimation bias, but we lack an understanding of how bias interacts with performance, and the extent to which existing algorithms mitigate bias. In this paper, we 1) highlight that the effect of overestimation bias on learning efficiency is environment-dependent; 2) propose a generalization of Q-learning, called Maxmin Q-learning, which provides a parameter to flexibly control bias; 3) show theoretically that there exists a parameter choice for Maxmin Q-learning that leads to unbiased estimation with a lower approximation variance than Q-learning; and 4) prove the convergence of our algorithm in the tabular case, as well as convergence of several previous Q-learning variants, using a novel Generalized Q-learning framework. We empirically verify that our algorithm better controls estimation bias in toy environments, and that it achieves superior performance on several benchmark problems.

Key facts. Cited by 117 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 27; 2022: 15; 2023: 18; 2024: 30; 2025: 17; 2026: 9.

117 papers.

Sort:

By area:

  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  2. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  3. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  4. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  5. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 52
  6. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  7. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  8. 2021Ensemble Bootstrapping for Q-Learningcited by 31
  9. 2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27
  10. 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 23
  11. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 21
  12. 2024Mixture of Experts in a Mixture of RL settingscited by 21
38 more
  1. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  2. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  3. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  4. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  5. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  6. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 17
  7. 2022Regularized Q-learningcited by 16
  8. 2023Ensemble Reinforcement Learning: A Surveycited by 16
  9. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  10. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13
  11. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  12. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  13. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  14. 2022Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Taskscited by 9
  15. 2022Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidationcited by 9
  16. 2023Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedbackcited by 9
  17. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
  18. 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8
  19. 2022Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue Systemcited by 8
  20. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learningcited by 8
  21. 2021Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performancecited by 7
  22. 2024Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equationcited by 7
  23. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learningcited by 7
  24. 2021Adaptively Calibrated Critic Estimates for Deep Reinforcement Learningcited by 6
  25. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  26. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  27. 2024Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learningcited by 6
  28. 2024SAPIENT: Mastering Multi-turn Conversational Recommendation with Strategic Planning and Monte Carlo Tree Searchcited by 6
  29. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 6
  30. 2021MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learningcited by 5
  31. 2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasetscited by 5
  32. 2022Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworkscited by 5
  33. 2023Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Controlcited by 5
  34. 2024Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimizationcited by 5
  35. 2024CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Criticscited by 5
  36. 2024Stochastic Q-learning for Large Discrete Action Spacescited by 5
  37. 2024Contextualized Hybrid Ensemble Q-learning: Learning Fast with Control Priorscited by 5
  38. 2025FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learningcited by 5

50 most cited of 117 All 117 (JSON)