mlsanityData to 7 Oct 2026

Snapshot Reinforcement Learning: Leveraging Prior Trajectories for Efficiency

arXiv 2403.00673 ↗First posted 1 Mar 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (DRL) algorithms require substantial samples and computational resources to achieve higher performance, which restricts their practical application and poses challenges for further development. Given the constraint of limited resources, it is essential to leverage existing computational work (e.g., learned policies, samples) to enhance sample efficiency and reduce the computational resource consumption of DRL algorithms. Previous works to leverage existing computational work require intrusive modifications to existing algorithms and models, designed specifically for specific algorithms, lacking flexibility and universality. In this paper, we present the Snapshot Reinforcement Learning (SnapshotRL) framework, which enhances sample efficiency by simply altering environments, without making any modifications to algorithms and models. By allowing student agents to choose states in teacher trajectories as the initial state to sample, SnapshotRL can effectively utilize teacher trajectories to assist student agents in training, allowing student agents to explore a larger state space at the early training phase. We propose a simple and effective SnapshotRL baseline algorithm, S3RL, which integrates well with existing DRL algorithms. Our experiments demonstrate that integrating S3RL with TD3, SAC, and PPO algorithms on the MuJoCo benchmark significantly improves sample efficiency and average return, without extra samples and additional computational resources.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

19 of 26 references.

References · 19

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  6. 2022Jump-Start Reinforcement Learningcited by 102
  7. 2019Distilling Policy Distillationcited by 81
  8. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  9. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  10. 2017Time Limits in Reinforcement Learningcited by 72
  11. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  12. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 43
7 more
  1. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutionscited by 26
  3. 2016Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replaycited by 22
  4. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learningcited by 17
  5. 2023Guiding Online Reinforcement Learning with Action-Free Offline Pretrainingcited by 11
  6. 2023Contrastive Initial State Buffer for Reinforcement Learningcited by 5
  7. 2018Sample-Efficient Learning of Nonprehensile Manipulation Policies via Physics-Based Informed State Distributionscited by 2

Built on by

Not cited yet.