# TD-Regularized Actor-Critic Methods

arXiv: https://arxiv.org/abs/1812.08288 · HTML: https://mlsanity.com/paper/1812.08288/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2018.

## Built on by (most cited first, 14 of 14)

- [WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering](https://mlsanity.com/paper/2512.10962/) (2025)
- [Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization](https://mlsanity.com/paper/2105.12991/) (2021)
- [Forward and inverse reinforcement learning sharing network weights and hyperparameters](https://mlsanity.com/paper/2008.07284/) (2020)
- [Reinforcement Learning Your Way: Agent Characterization through Policy Regularization](https://mlsanity.com/paper/2201.10003/) (2022)
- [Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio](https://mlsanity.com/paper/2203.09809/) (2022)
- [Reward Bonuses with Gain Scheduling Inspired by Iterative Deepening Search](https://mlsanity.com/paper/2212.10765/) (2022)
- [KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation](https://mlsanity.com/paper/2504.19024/) (2025)
- [Temporal Difference Learning with Constrained Initial Representations](https://mlsanity.com/paper/2602.11800/) (2026)
- [Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency](https://mlsanity.com/paper/2412.12894/) (2024)
- [Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Search](https://mlsanity.com/paper/2509.15927/) (2025)
- [Task Specific Sharpness Aware O-RAN Resource Management using Multi Agent Reinforcement Learning](https://mlsanity.com/paper/2511.15002/) (2025)
- [General sum stochastic games with networked information flows](https://mlsanity.com/paper/2205.02760/) (2022)
- [Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning](https://mlsanity.com/paper/2311.03711/) (2023)
- [Input-Side Variance Suppression under Non-Normal Transient Amplification in Continuous-Control Reinforcement Learning](https://mlsanity.com/paper/2604.17744/) (2026)

## Abstract (arXiv)

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize the learning. To avoid such issues, we propose to regularize the learning objective of the actor by penalizing the temporal difference (TD) error of the critic. This improves stability by avoiding large steps in the actor update whenever the critic is highly inaccurate. The resulting method, which we call the TD-regularized actor-critic method, is a simple plug-and-play approach to improve stability and overall performance of the actor-critic methods. Evaluations on standard benchmarks confirm this.