Learning the Target Network in Function Space
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
We focus on the task of learning the value function in the reinforcement learning (RL) setting. This task is often solved by updating a pair of online and target networks while ensuring that the parameters of these two networks are equivalent. We propose Lookahead-Replicate (LR), a new value-function approximation algorithm that is agnostic to this parameter-space equivalence. Instead, the LR algorithm is designed to maintain an equivalence between the two networks in the function space. This value-based equivalence is obtained by employing a new target-network update. We show that LR leads to a convergent behavior in learning the value function. We also present empirical results demonstrating that LR-based target-network updates significantly improve deep RL on the Atari benchmark.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 52 references that are in the map. First posted on arXiv 3 Jun 2024.
Builds on
27 of 52 references.
References · 27
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2015Trust Region Policy Optimizationcited by 2,580
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2015Policy Distillationcited by 440
- 2018Maximum a Posteriori Policy Optimisationcited by 328
15 more
- 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
- 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
- 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
- 2016Learning values across many orders of magnitudecited by 85
- 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
- 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 65
- 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57
- 2021Breaking the Deadly Triad with a Target Networkcited by 47
- 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
- 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
- 2019Target-Based Temporal Difference Learningcited by 20
- 2023TD Convergence: An Optimization Perspectivecited by 16
- 2020Regularized Off-Policy TD-Learningcited by 10
- 2021Faster Deep Reinforcement Learning with Slower Online Networkcited by 7
- 2018Natural Gradient Deep Q-learningcited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
- 2026Target-Aligned Reinforcement Learning
- 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
- 2025Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learning
- 2026Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation
- 2019On Value Functions and the Agent-Environment Boundary
- 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning
- 2025A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning