Data to 7 Oct 2026

Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

arXiv 2005.04269 ↗First posted 8 May 2020cs.LG cs.AI stat.MLcited by 154

Not on the map. Why?

Abstract (arXiv)

The overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous control setting. Our method---Truncated Quantile Critics, TQC,---blends three ideas: distributional representation of a critic, truncation of critics prediction, and ensembling of multiple critics. Distributional representation and truncation allow for arbitrary granular overestimation control, while ensembling provides additional score improvements. TQC outperforms the current state of the art on all environments from the continuous control benchmark suite, demonstrating 25% improvement on the most challenging Humanoid environment.

Key facts. Cited by 154 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 19; 2023: 30; 2024: 31; 2025: 27; 2026: 24.

154 papers.

Sort:

By area:

2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
2022AKB-48: A Real-World Articulated Object Knowledge Basecited by 81
2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 38
2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27

38 more

2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 22
2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
2022Low-Rank Modular Reinforcement Learning via Muscle Synergycited by 16
2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
2021GMAC: A Distributional Perspective on Actor-Critic Frameworkcited by 15
2024DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulationcited by 15
2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
2023Testing of Deep Reinforcement Learning Agents with Surrogate Modelscited by 12
2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
2023Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learningcited by 12
2021MDPFuzz: Testing Models Solving Markov Decision Processescited by 11
2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
2025Neural Motion Simulator: Pushing the Limit of World Models in Reinforcement Learningcited by 11
2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
2022Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probabilitycited by 9
2022Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue Systemcited by 8
2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
2021Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performancecited by 7
2023RL-X: A Deep Reinforcement Learning Library (not only) for RoboCupcited by 7
2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
2024Efficient Active Imitation Learning with Random Network Distillationcited by 7
2021Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradientscited by 6
2022SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Criticscited by 6
2022Leveraging Sequentiality in Reinforcement Learning from a Single Demonstrationcited by 6
2023A Circuit Domain Generalization Framework for Efficient Logic Synthesis in Chip Designcited by 6
2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6
2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasetscited by 5
2021Adaptively Calibrated Critic Estimates for Deep Reinforcement Learningcited by 5

50 most cited of 154 All 154 (JSON)