# Learning to Sample with Local and Global Contexts in Experience Replay Buffer

arXiv: https://arxiv.org/abs/2007.07358 · HTML: https://mlsanity.com/paper/2007.07358/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [Actor Prioritized Experience Replay](https://mlsanity.com/paper/2209.00532/) (2022)
- [Curious Replay for Model-based Adaptation](https://mlsanity.com/paper/2306.15934/) (2023)
- [Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.15612/) (2022)
- [Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replay](https://mlsanity.com/paper/2111.01865/) (2021)
- [Reliability-Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2506.18482/) (2025)
- [CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/2406.09030/) (2024)
- [Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards](https://mlsanity.com/paper/2311.00426/) (2023)
- [Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiences](https://mlsanity.com/paper/2402.05963/) (2024)
- [Enhancing Deep Deterministic Policy Gradients on Continuous Control Tasks with Decoupled Prioritized Experience Replay](https://mlsanity.com/paper/2512.05320/) (2025)
- [Uncertainty-Driven Replay Memory for Reinforcement Learning](https://mlsanity.com/paper/2608.29860/) (2026)

## Abstract (arXiv)

Experience replay, which enables the agents to remember and reuse experience from the past, has played a significant role in the success of off-policy reinforcement learning (RL). To utilize the experience replay efficiently, the existing sampling methods allow selecting out more meaningful experiences by imposing priorities on them based on certain metrics (e.g. TD-error). However, they may result in sampling highly biased, redundant transitions since they compute the sampling rate for each transition independently, without consideration of its importance in relation to other transitions. In this paper, we aim to address the issue by proposing a new learning-based sampling method that can compute the relative importance of transition. To this end, we design a novel permutation-equivariant neural architecture that takes contexts from not only features of each transition (local) but also those of others (global) as inputs. We validate our framework, which we refer to as Neural Experience Replay Sampler (NERS), on multiple benchmark tasks for both continuous and discrete control tasks and show that it can significantly improve the performance of various off-policy RL methods. Further analysis confirms that the improvements of the sample efficiency indeed are due to sampling diverse and meaningful transitions by NERS that considers both local and global contexts.