# Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning

arXiv: https://arxiv.org/abs/2409.19231 · HTML: https://mlsanity.com/paper/2409.19231/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [For SALE: State-Action Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2306.02451/) (2023)
- [How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments](https://mlsanity.com/paper/1806.08295/) (2018)
- [Softmax Deep Double Deterministic Policy Gradients](https://mlsanity.com/paper/2010.09177/) (2020)
- [Distributional Soft Actor-Critic with Three Refinements](https://mlsanity.com/paper/2310.05858/) (2023)
- [Efficient Continuous Control with Double Actors and Regularized Critics](https://mlsanity.com/paper/2106.03050/) (2021)
- [Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration](https://mlsanity.com/paper/2006.01419/) (2020)
- [Deterministic Implementations for Reproducibility in Deep Reinforcement Learning](https://mlsanity.com/paper/1809.05676/) (2018)
- [Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2110.03375/) (2021)
- [Deep Conservative Policy Iteration](https://mlsanity.com/paper/1906.09784/) (2019)
- [Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian](https://mlsanity.com/paper/2108.11623/) (2021)
- [Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments](https://mlsanity.com/paper/1903.00827/) (2019)
- [Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Tasks](https://mlsanity.com/paper/2203.11526/) (2022)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning](https://mlsanity.com/paper/2401.03137/) (2024)
- [Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradients](https://mlsanity.com/paper/2112.11216/) (2021)
- [Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control](https://mlsanity.com/paper/2310.11138/) (2023)
- [ACERAC: Efficient reinforcement learning in fine time discretization](https://mlsanity.com/paper/2104.04004/) (2021)
- [On the Estimation Bias in Double Q-Learning](https://mlsanity.com/paper/2109.14419/) (2021)
- [Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach](https://mlsanity.com/paper/2208.00755/) (2022)

## Built on by (most cited first, 3 of 3)

- [Wasserstein Barycenter Soft Actor-Critic](https://mlsanity.com/paper/2506.10167/) (2025)
- [Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularization](https://mlsanity.com/paper/2511.16090/) (2025)
- [A Cognitive Framework for Autonomous Agents: Toward Human-Inspired Design](https://mlsanity.com/paper/2601.16648/) (2026)

## Abstract (arXiv)

To obtain better value estimation in reinforcement learning, we propose a novel algorithm based on the double actor-critic framework with temporal difference error-driven regularization, abbreviated as TDDR. TDDR employs double actors, with each actor paired with a critic, thereby fully leveraging the advantages of double critics. Additionally, TDDR introduces an innovative critic regularization architecture. Compared to classical deterministic policy gradient-based algorithms that lack a double actor-critic structure, TDDR provides superior estimation. Moreover, unlike existing algorithms with double actor-critic frameworks, TDDR does not introduce any additional hyperparameters, significantly simplifying the design and implementation process. Experiments demonstrate that TDDR exhibits strong competitiveness compared to benchmark algorithms in challenging continuous control tasks.