mlsanityData to 14 Oct 2026

Replay across Experiments: A Natural Extension of Off-Policy RL

arXiv 2311.15951 ↗First posted 27 Nov 2023cs.LG cs.AI cs.ROcited by 10

Not on the map. Why?

Abstract (arXiv)

Replaying data is a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning (RL). We present an effective yet simple framework to extend the use of replays across multiple experiments, minimally adapting the RL workflow for sizeable improvements in controller performance and research iteration times. At its core, Replay Across Experiments (RaE) involves reusing experience from previous experiments to improve exploration and bootstrap learning while reducing required changes to a minimum in comparison to prior work. We empirically show benefits across a number of RL algorithms and challenging control domains spanning both locomotion and manipulation, including hard exploration tasks from egocentric vision. Through comprehensive ablations, we demonstrate robustness to the quality and amount of data available and various hyperparameter choices. Finally, we discuss how our approach can be applied more broadly across research life cycles and can increase resilience by reloading data across random seeds or hyperparameter variations.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

31 of 44 references.

References · 31

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
19 more
  1. 2020Critic Regularized Regressioncited by 275
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  5. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  6. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  7. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  8. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  9. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  10. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  11. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  12. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 71
  13. 2019Simulation-based reinforcement learning for real-world autonomous drivingcited by 52
  14. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 33
  15. 2022Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviorscited by 30
  16. 2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 29
  17. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  18. 2020Learning Dexterous Manipulation from Suboptimal Expertscited by 23
  19. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 2.

10 papers.

  1. 2024Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learningcited by 17
  2. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  3. 2024Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learningcited by 5
  4. 2025Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformerscited by 4
  5. 2025Automated Reward Design for Gran Turismocited by 3
  6. 2026What Matters for Simulation to Online Reinforcement Learning on Real Robotscited by 3
  7. 2025Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approachcited by 2
  8. 2024Real-World Fluid Directed Rigid Body Control via Deep Reinforcement Learningcited by 1
  9. 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
  10. 2026Augmenting Game AI with Deep Reinforcement Learningcited by 0