mlsanityData to 14 Oct 2026

Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularization

arXiv 2511.16090 ↗First posted 20 Nov 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Deterministic policy gradient algorithms for continuous control suffer from value estimation biases that degrade performance. While double critics reduce such biases, the exploration potential of double actors remains underexplored. Building on temporal-difference error-driven regularization (TDDR), a double actor-critic framework, this work introduces enhanced methods to achieve flexible bias control and stronger representation learning. We propose three convex combination strategies, symmetric and asymmetric, that balance pessimistic estimates to mitigate overestimation and optimistic exploration via double actors to alleviate underestimation. A single hyperparameter governs this mechanism, enabling tunable control across the bias spectrum. To further improve performance, we integrate augmented state and action representations into the actor and critic networks. Extensive experiments show that our approach consistently outperforms benchmarks, demonstrating the value of tunable bias and revealing that both overestimation and underestimation can be exploited differently depending on the environment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 45 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

30 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning28 Sep
  2. 2024Symmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning12 Mar

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2016OpenAI Gymcited by 1,736
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  9. 2020Reinforcement Learning with Augmented Datacited by 468
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  12. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
16 more
  1. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  2. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  3. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  4. 2020Softmax Deep Double Deterministic Policy Gradientscited by 32
  5. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  6. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 21
  7. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  8. 2023ReCoRe: Regularized Contrastive Representation Learning of World Modelcited by 11
  9. 2022Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Taskscited by 9
  10. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learningcited by 8
  11. 2021Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradientscited by 6
  12. 2023Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Controlcited by 5
  13. 2021ACERAC: Efficient reinforcement learning in fine time discretizationcited by 4
  14. 2021On the Estimation Bias in Double Q-Learningcited by 4
  15. 2022Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approachcited by 3
  16. 2021Automating Control of Overestimation Bias for Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks
  2. 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods
  3. 2026Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
  4. 2025Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning
  5. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty
  6. 2025Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss
  7. 2024Compatible Gradient Approximations for Actor-Critic Algorithms
  8. 2026Refined Analysis of Entropy-Regularized Actor-Critic