# Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradients

arXiv: https://arxiv.org/abs/2112.11216 · HTML: https://mlsanity.com/paper/2112.11216/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 26 of 26)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics](https://mlsanity.com/paper/2005.04269/) (2020)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [Continuous Deep Q-Learning with Model-based Acceleration](https://mlsanity.com/paper/1603.00748/) (2016)
- [Better Exploration with Optimistic Actor-Critic](https://mlsanity.com/paper/1910.12807/) (2019)
- [Chip Placement with Deep Reinforcement Learning](https://mlsanity.com/paper/2004.10746/) (2020)
- [Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.03400/) (2021)
- [Towards Playing Full MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.12692/) (2020)
- [Data-efficient Deep Reinforcement Learning for Dexterous Manipulation](https://mlsanity.com/paper/1704.03073/) (2017)
- [Tactical Optimism and Pessimism for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.03765/) (2021)
- [AlphaStar: An Evolutionary Computation Perspective](https://mlsanity.com/paper/1902.01724/) (2019)
- [Offline Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.01832/) (2021)
- [Softmax Deep Double Deterministic Policy Gradients](https://mlsanity.com/paper/2010.09177/) (2020)
- [Adaptive Power System Emergency Control using Deep Reinforcement Learning](https://mlsanity.com/paper/1903.03712/) (2019)
- [Efficient Continuous Control with Double Actors and Regularized Critics](https://mlsanity.com/paper/2106.03050/) (2021)
- [The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning](https://mlsanity.com/paper/2006.12692/) (2020)
- [DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2004.14547/) (2020)

## Built on by (most cited first, 6 of 6)

- [Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse](https://mlsanity.com/paper/2305.18443/) (2023)
- [Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning](https://mlsanity.com/paper/2409.19231/) (2024)
- [Temporal Difference Learning with Constrained Initial Representations](https://mlsanity.com/paper/2602.11800/) (2026)
- [Wasserstein Barycenter Soft Actor-Critic](https://mlsanity.com/paper/2506.10167/) (2025)
- [Debiased Model-based Representations for Sample-efficient Continuous Control](https://mlsanity.com/paper/2605.11711/) (2026)
- [Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularization](https://mlsanity.com/paper/2511.16090/) (2025)

## Abstract (arXiv)

It is vital to accurately estimate the value function in Deep Reinforcement Learning (DRL) such that the agent could execute proper actions instead of suboptimal ones. However, existing actor-critic methods suffer more or less from underestimation bias or overestimation bias, which negatively affect their performance. In this paper, we reveal a simple but effective principle: proper value correction benefits bias alleviation, where we propose the generalized-activated weighting operator that uses any non-decreasing function, namely activation function, as weights for better value estimation. Particularly, we integrate the generalized-activated weighting operator into value estimation and introduce a novel algorithm, Generalized-activated Deep Double Deterministic Policy Gradients (GD3). We theoretically show that GD3 is capable of alleviating the potential estimation bias. We interestingly find that simple activation functions lead to satisfying performance with no additional tricks, and could contribute to faster convergence. Experimental results on numerous challenging continuous control tasks show that GD3 with task-specific activation outperforms the common baseline methods. We also uncover a fact that fine-tuning the polynomial activation function achieves superior results on most of the tasks.