mlsanityData to 14 Oct 2026

Novelty-based Sample Reuse for Continuous Robotics Control

arXiv 2410.13490 ↗First posted 17 Oct 2024cs.RO cs.LGcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In reinforcement learning, agents collect state information and rewards through environmental interactions, essential for policy refinement. This process is notably time-consuming, especially in complex robotic simulations and real-world applications. Traditional algorithms usually re-engage with the environment after processing a single batch of samples, thereby failing to fully capitalize on historical data. However, frequently observed states, with reliable value estimates, require minimal updates; in contrast, rare observed states necessitate more intensive updates for achieving accurate value estimations. To address uneven sample utilization, we propose Novelty-guided Sample Reuse (NSR). NSR provides extra updates for infrequent, novel states and skips additional updates for frequent states, maximizing sample use before interacting with the environment again. Our experiments show that NSR improves the convergence rate and success rate of algorithms without significantly increasing time consumption. Our code is publicly available at https://github.com/ppksigs/NSR-DDPG-HER.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 38 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

34 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Exploration and Anti-Exploration with Distributional Random Network Distillation18 Jan

Other references · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2016Progressive Neural Networkscited by 1,361
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  9. 2017Hindsight Experience Replaycited by 1,136
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  12. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
21 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  4. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  5. 2019Model-Based Reinforcement Learning for Ataricited by 502
  6. 2017A Tutorial on Thompson Samplingcited by 433
  7. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  8. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  9. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  10. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  11. 2020First return, then explorecited by 266
  12. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  13. 2017Count-Based Exploration with Neural Density Modelscited by 257
  14. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  15. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 178
  16. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  17. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  18. 2018Count-Based Exploration with the Successor Representationcited by 112
  19. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101
  20. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  21. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control
  2. 2025CCDP: Composition of Conditional Diffusion Policies with Guided Sampling
  3. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement
  4. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation
  5. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
  6. 2024State-Novelty Guided Action Persistence in Deep Reinforcement Learning
  7. 2025Reinforcement Learning via Implicit Imitation Guidance
  8. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning