mlsanityData to 14 Oct 2026

Consolidated Adaptive T-soft Update for Deep Reinforcement Learning

arXiv 2202.12504 ↗First posted 25 Feb 2022cs.LG cs.ROcited by 5

Not on the map. Why?

Abstract (arXiv)

Demand for deep reinforcement learning (DRL) is gradually increased to enable robots to perform complex tasks, while DRL is known to be unstable. As a technique to stabilize its learning, a target network that slowly and asymptotically matches a main network is widely employed to generate stable pseudo-supervised signals. Recently, T-soft update has been proposed as a noise-robust update rule for the target network and has contributed to improving the DRL performance. However, the noise robustness of T-soft update is specified by a hyperparameter, which should be tuned for each task, and is deteriorated by a simplified implementation. This study develops adaptive T-soft (AT-soft) update by utilizing the update rule in AdaTerm, which has been developed recently. In addition, the concern that the target network does not asymptotically match the main network is mitigated by a new consolidation for bringing the main network back to the target network. This so-called consolidated AT-soft (CAT-soft) update is verified through numerical simulations.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 25 Feb 2022.

Builds on

19 of 40 references.

References · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2016OpenAI Gymcited by 1,736
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  9. 2016Deep Exploration via Bootstrapped DQNcited by 527
  10. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  11. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 142
  12. 2021Breaking the Deadly Triad with a Target Networkcited by 47
7 more
  1. 2022L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learningcited by 27
  2. 2020PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inferencecited by 23
  3. 2020t-Soft Update of Target Network for Deep Reinforcement Learningcited by 17
  4. 2021Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcited by 14
  5. 2022AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimizationcited by 11
  6. 2020Proximal Policy Optimization with Relative Pearson Divergencecited by 9
  7. 2021Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 0; 2026: 1.

5 papers.

  1. 2022Reward Bonuses with Gain Scheduling Inspired by Iterative Deepening Searchcited by 5
  2. 2023Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraintcited by 3
  3. 2024Revisiting Experience Replayable Conditionscited by 3
  4. 2026Flexible Empowerment at Reasoning with Extended Best-of-N Samplingcited by 3
  5. 2024Weber-Fechner Law in Temporal Difference learning derived from Control as Inferencecited by 1