# Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2410.20487 · HTML: https://mlsanity.com/paper/2410.20487/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 37 of its 56 references that are in the map.
- First posted on arXiv 27 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 37)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [An Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/1811.12560/) (2018)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments](https://mlsanity.com/paper/1612.05533/) (2016)
- [Discriminator Rejection Sampling](https://mlsanity.com/paper/1810.06758/) (2018)
- [Diversity driven Attention Model for Query-based Abstractive Summarization](https://mlsanity.com/paper/1704.08300/) (2017)
- [Dealing with Sparse Rewards in Reinforcement Learning](https://mlsanity.com/paper/1910.09281/) (2019)
- [On the Importance of Exploration for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2306.05483/) (2023)
- [Prioritizing Samples in Reinforcement Learning with Reducible Loss](https://mlsanity.com/paper/2208.10483/) (2022)
- [Energy-Based Hindsight Experience Prioritization](https://mlsanity.com/paper/1810.01363/) (2018)
- [Actor Prioritized Experience Replay](https://mlsanity.com/paper/2209.00532/) (2022)

## Built on by (most cited first, 7 of 7)

- [Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs](https://mlsanity.com/paper/2512.06258/) (2025)
- [MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting](https://mlsanity.com/paper/2601.09085/) (2026)
- [UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMs](https://mlsanity.com/paper/2602.22296/) (2026)
- [PER-DPP Sampling Framework and Its Application in Path Planning](https://mlsanity.com/paper/2503.07411/) (2025)
- [C2T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordination](https://mlsanity.com/paper/2604.13098/) (2026)
- [E2DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation](https://mlsanity.com/paper/2605.00159/) (2026)
- [Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning](https://mlsanity.com/paper/2607.23474/) (2026)

## Abstract (arXiv)

Experience replay is widely used to improve learning efficiency in reinforcement learning by leveraging past experiences. However, existing experience replay methods, whether based on uniform or prioritized sampling, often suffer from low efficiency, particularly in real-world scenarios with high-dimensional state spaces. To address this limitation, we propose a novel approach, Efficient Diversity-based Experience Replay (EDER). EDER employs a determinantal point process to model the diversity between samples and prioritizes replay based on the diversity between samples. To further enhance learning efficiency, we incorporate Cholesky decomposition for handling large state spaces in realistic environments. Additionally, rejection sampling is applied to select samples with higher diversity, thereby improving overall learning efficacy. Extensive experiments are conducted on robotic manipulation tasks in MuJoCo, Atari games, and realistic indoor environments in Habitat. The results demonstrate that our approach not only significantly improves learning efficiency but also achieves superior performance in high-dimensional, realistic environments.