mlsanityData to 7 Oct 2026

Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse

arXiv 2305.18443 ↗First posted 29 May 2023cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Sample efficiency is one of the most critical issues for online reinforcement learning (RL). Existing methods achieve higher sample efficiency by adopting model-based methods, Q-ensemble, or better exploration mechanisms. We, instead, propose to train an off-policy RL agent via updating on a fixed sampled batch multiple times, thus reusing these samples and better exploiting them within a single optimization loop. We name our method sample multiple reuse (SMR). We theoretically show the properties of Q-learning with SMR, e.g., convergence. Furthermore, we incorporate SMR with off-the-shelf off-policy RL algorithms and conduct experiments on a variety of continuous control benchmarks. Empirical results show that SMR significantly boosts the sample efficiency of the base methods across most of the evaluated tasks without any hyperparameter tuning or additional tricks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 71 of its 112 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

71 of 112 references.

References · 71

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2016OpenAI Gymcited by 1,684
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  12. 2015Prioritized Experience Replaycited by 1,070
38 more
  1. 2020Mastering Atari with Discrete World Modelscited by 990
  2. 2018Exploration by Random Network Distillationcited by 966
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  4. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2017Deep Reinforcement Learning that Matterscited by 727
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2019Model-Based Reinforcement Learning for Ataricited by 495
  12. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  13. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  14. 2020Reinforcement Learning with Augmented Datacited by 452
  15. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  16. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  17. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  18. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  19. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  20. 2020First return, then explorecited by 256
  21. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  22. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  23. 2021Mastering Atari Games with Limited Datacited by 235
  24. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  25. 2020Revisiting Fundamentals of Experience Replaycited by 171
  26. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  27. 2017Parameter Space Noise for Explorationcited by 158
  28. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  29. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  30. 2019When to use parametric models in reinforcement learning?cited by 118
  31. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  32. 2017A Deeper Look at Experience Replaycited by 104
  33. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 93
  34. 2020Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncited by 93
  35. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  36. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  37. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  38. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 4; 2026: 4.

12 papers.

  1. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  2. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  3. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  4. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
  5. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learningcited by 7
  6. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidencecited by 5
  7. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  8. 2024Novelty-based Sample Reuse for Continuous Robotics Controlcited by 1
  9. 2026SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Datacited by 1
  10. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  11. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  12. 2025Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Methodcited by 0