# Target-Based Temporal Difference Learning

arXiv: https://arxiv.org/abs/1904.10945 · HTML: https://mlsanity.com/paper/1904.10945/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2019.

## Built on by (most cited first, 20 of 20)

- [TD Convergence: An Optimization Perspective](https://mlsanity.com/paper/2306.17750/) (2023)
- [Rethinking ValueDice: Does It Really Improve Performance?](https://mlsanity.com/paper/2202.02468/) (2022)
- [Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximation](https://mlsanity.com/paper/2106.07472/) (2021)
- [Why Target Networks Stabilise Temporal Difference Methods](https://mlsanity.com/paper/2302.12537/) (2023)
- [Improved Training Technique for Latent Consistency Models](https://mlsanity.com/paper/2502.01441/) (2025)
- [Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation](https://mlsanity.com/paper/2405.21043/) (2024)
- [Faster Deep Reinforcement Learning with Slower Online Network](https://mlsanity.com/paper/2112.05848/) (2021)
- [Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features](https://mlsanity.com/paper/2409.12135/) (2024)
- [A Note on Target Q-learning For Solving Finite MDPs with A Generative Oracle](https://mlsanity.com/paper/2203.11489/) (2022)
- [Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics](https://mlsanity.com/paper/2606.02645/) (2026)
- [Finite-Time Analysis of Temporal Difference Learning with Experience Replay](https://mlsanity.com/paper/2306.09746/) (2023)
- [Learning the Target Network in Function Space](https://mlsanity.com/paper/2406.01838/) (2024)
- [Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approach](https://mlsanity.com/paper/2410.13463/) (2024)
- [Periodic Regularized Q-Learning](https://mlsanity.com/paper/2602.03301/) (2026)
- [Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error](https://mlsanity.com/paper/2604.01613/) (2026)
- [Control Theoretic Analysis of Temporal Difference Learning](https://mlsanity.com/paper/2112.14417/) (2021)
- [Online Attentive Kernel-Based Temporal Difference Learning](https://mlsanity.com/paper/2201.09065/) (2022)
- [Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Exploration](https://mlsanity.com/paper/2509.22964/) (2025)
- [Geometrically Averaged Hard Target Updates for Linear Q-Learning](https://mlsanity.com/paper/2606.10835/) (2026)
- [Revisiting TD Target Aggregation under Uncertainty in Q-Learning](https://mlsanity.com/paper/2608.03069/) (2026)

## Abstract (arXiv)

The use of target networks has been a popular and key component of recent deep Q-learning algorithms for reinforcement learning, yet little is known from the theory side. In this work, we introduce a new family of target-based temporal difference (TD) learning algorithms and provide theoretical analysis on their convergences. In contrast to the standard TD-learning, target-based TD algorithms maintain two separate learning parameters-the target variable and online variable. Particularly, we introduce three members in the family, called the averaging TD, double TD, and periodic TD, where the target variable is updated through an averaging, symmetric, or periodic fashion, mirroring those techniques used in deep Q-learning practice. We establish asymptotic convergence analyses for both averaging TD and double TD and a finite sample analysis for periodic TD. In addition, we also provide some simulation results showing potentially superior convergence of these target-based TD algorithms compared to the standard TD-learning. While this work focuses on linear function approximation and policy evaluation setting, we consider this as a meaningful step towards the theoretical understanding of deep Q-learning variants with target networks.