# t-Soft Update of Target Network for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2008.10861 · HTML: https://mlsanity.com/paper/2008.10861/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2020.

## Built on by (most cited first, 17 of 17)

- [L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning](https://mlsanity.com/paper/2202.07152/) (2022)
- [AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization](https://mlsanity.com/paper/2201.06714/) (2022)
- [Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization](https://mlsanity.com/paper/2105.12991/) (2021)
- [WISK: A Workload-aware Learned Index for Spatial Keyword Queries](https://mlsanity.com/paper/2302.14287/) (2023)
- [Consolidated Adaptive T-soft Update for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.12504/) (2022)
- [Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio](https://mlsanity.com/paper/2203.09809/) (2022)
- [Adaptive t-Momentum-based Optimization for Unknown Ratio of Outliers in Amateur Data in Imitation Learning](https://mlsanity.com/paper/2108.00625/) (2021)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [Cooperative Multi-Agent Deep Reinforcement Learning in Content Ranking Optimization](https://mlsanity.com/paper/2408.04251/) (2024)
- [Physics-Guided Reinforcement Learning System for Realistic Vehicle Active Suspension Control](https://mlsanity.com/paper/2408.08425/) (2024)
- [Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures](https://mlsanity.com/paper/2104.00385/) (2021)
- [Toward Debugging Deep Reinforcement Learning Programs with RLExplorer](https://mlsanity.com/paper/2410.04322/) (2024)
- [Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency](https://mlsanity.com/paper/2412.12894/) (2024)
- [Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error](https://mlsanity.com/paper/2604.01613/) (2026)
- [Cross Modality 3D Navigation Using Reinforcement Learning and Neural Style Transfer](https://mlsanity.com/paper/2111.03485/) (2021)
- [Achieving Hiding and Smart Anti-Jamming Communication: A Parallel DRL Approach against Moving Reactive Jammer](https://mlsanity.com/paper/2502.02385/) (2025)
- [Target-Aligned Reinforcement Learning](https://mlsanity.com/paper/2603.29501/) (2026)

## Abstract (arXiv)

This paper proposes a new robust update rule of target network for deep reinforcement learning (DRL), to replace the conventional update rule, given as an exponential moving average. The target network is for smoothly generating the reference signals for a main network in DRL, thereby reducing learning variance. The problem with its conventional update rule is the fact that all the parameters are smoothly copied with the same speed from the main network, even when some of them are trying to update toward the wrong directions. This behavior increases the risk of generating the wrong reference signals. Although slowing down the overall update speed is a naive way to mitigate wrong updates, it would decrease learning speed. To robustly update the parameters while keeping learning speed, a t-soft update method, which is inspired by student-t distribution, is derived with reference to the analogy between the exponential moving average and the normal distribution. Through the analysis of the derived t-soft update, we show that it takes over the properties of the student-t distribution. Specifically, with a heavy-tailed property of the student-t distribution, the t-soft update automatically excludes extreme updates that differ from past experiences. In addition, when the updates are similar to the past experiences, it can mitigate the learning delay by increasing the amount of updates. In PyBullet robotics simulations for DRL, an online actor-critic algorithm with the t-soft update outperformed the conventional methods in terms of the obtained return and/or its variance. From the training process by the t-soft update, we found that the t-soft update is globally consistent with the standard soft update, and the update rates are locally adjusted for acceleration or suppression.