mlsanityData to 7 Oct 2026

Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning

arXiv 2510.02590 ↗First posted 2 Oct 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

The use of target networks is a popular approach for estimating value functions in deep Reinforcement Learning (RL). While effective, the target network remains a compromise solution that preserves stability at the cost of slowly moving targets, thus delaying learning. Conversely, using the online network as a bootstrapped target is intuitively appealing, albeit well-known to lead to unstable learning. In this work, we aim to obtain the best out of both worlds by introducing a novel update rule that computes the target using the MINimum estimate between the Target and Online network, giving rise to our method, MINTO. Through this simple, yet effective modification, we show that MINTO enables faster and stable value function learning, by mitigating the potential overestimation bias of using the online network for bootstrapping. Notably, MINTO can be seamlessly integrated into a wide range of value-based and actor-critic algorithms with a negligible cost. We evaluate MINTO extensively across diverse benchmarks, spanning online and offline RL, as well as discrete and continuous action spaces. Across all benchmarks, MINTO consistently improves performance, demonstrating its broad applicability and effectiveness.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

37 of 53 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2018DeepMind Control Suitecited by 935
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
25 more
  1. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  2. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  3. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  4. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  5. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  6. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  7. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  8. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  9. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  10. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  11. 2024Simplifying Deep Temporal Difference Learningcited by 71
  12. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  13. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  14. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  15. 2022Human-level Atari 200x fastercited by 28
  16. 2020GRAC: Self-Guided and Self-Regularized Actor-Criticcited by 20
  17. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  18. 2021Functional Regularization for Reinforcement Learning via Learned Fourier Featurescited by 17
  19. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  20. 2020Self-correcting Q-Learningcited by 9
  21. 2021Bridging the Gap Between Target Networks and Functional Regularizationcited by 7
  22. 2020Stabilizing Q Learning Via Soft Mellowmax Operatorcited by 6
  23. 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learningcited by 6
  24. 2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learningcited by 6
  25. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  2. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning
  2. 2026Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
  3. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  4. 2022A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions
  5. 2026VIMPO: Value-Implicit Policy Optimization for LLMs
  6. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
  7. 2025EXPO: Stable Reinforcement Learning with Expressive Policies
  8. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning