# Dynamic Experience Replay

arXiv: https://arxiv.org/abs/2003.02372 · HTML: https://mlsanity.com/paper/2003.02372/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2020.

## Built on by (most cited first, 7 of 7)

- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)
- [JUICER: Data-Efficient Imitation Learning for Robotic Assembly](https://mlsanity.com/paper/2404.03729/) (2024)
- [Minimizing Human Assistance: Augmenting a Single Demonstration for Deep Reinforcement Learning](https://mlsanity.com/paper/2209.11275/) (2022)
- [Using Implicit Behavior Cloning and Dynamic Movement Primitive to Facilitate Reinforcement Learning for Robot Motion Planning](https://mlsanity.com/paper/2307.16062/) (2023)
- [Transformer with Memory Replay](https://mlsanity.com/paper/2205.09869/) (2022)
- [Continual Learning from Simulated Interactions via Multitask Prospective Rehearsal for Bionic Limb Behavior Modeling](https://mlsanity.com/paper/2405.01114/) (2024)
- [Uncertainty-Driven Replay Memory for Reinforcement Learning](https://mlsanity.com/paper/2608.29860/) (2026)

## Abstract (arXiv)

We present a novel technique called Dynamic Experience Replay (DER) that allows Reinforcement Learning (RL) algorithms to use experience replay samples not only from human demonstrations but also successful transitions generated by RL agents during training and therefore improve training efficiency. It can be combined with an arbitrary off-policy RL algorithm, such as DDPG or DQN, and their distributed versions. We build upon Ape-X DDPG and demonstrate our approach on robotic tight-fitting joint assembly tasks, based on force/torque and Cartesian pose observations. In particular, we run experiments on two different tasks: peg-in-hole and lap-joint. In each case, we compare different replay buffer structures and how DER affects them. Our ablation studies show that Dynamic Experience Replay is a crucial ingredient that either largely shortens the training time in these challenging environments or solves the tasks that the vanilla Ape-X DDPG cannot solve. We also show that our policies learned purely in simulation can be deployed successfully on the real robot. The video presenting our experiments is available at https://sites.google.com/site/dynamicexperiencereplay