# Prioritized Sequence Experience Replay

arXiv: https://arxiv.org/abs/1905.12726 · HTML: https://mlsanity.com/paper/1905.12726/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2019.

## Built on by (most cited first, 20 of 20)

- [Regret Minimization Experience Replay in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2105.07253/) (2021)
- [Actor Prioritized Experience Replay](https://mlsanity.com/paper/2209.00532/) (2022)
- [MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization](https://mlsanity.com/paper/2302.10418/) (2023)
- [Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy](https://mlsanity.com/paper/2207.12141/) (2022)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [Revisiting Prioritized Experience Replay: A Value Perspective](https://mlsanity.com/paper/2102.03261/) (2021)
- [From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks](https://mlsanity.com/paper/2201.08300/) (2022)
- [Decoupled Prioritized Resampling for Offline RL](https://mlsanity.com/paper/2306.05412/) (2023)
- [State-Aware Variational Thompson Sampling for Deep Q-Networks](https://mlsanity.com/paper/2102.03719/) (2021)
- [Active Advantage-Aligned Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2502.07937/) (2025)
- [Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments](https://mlsanity.com/paper/2207.01251/) (2022)
- [Reliability-Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2506.18482/) (2025)
- [Sample Efficient Experience Replay in Non-stationary Environments](https://mlsanity.com/paper/2509.15032/) (2025)
- [Hindsight States: Blending Sim and Real Task Elements for Efficient Reinforcement Learning](https://mlsanity.com/paper/2303.02234/) (2023)
- [Model-Augmented Q-learning](https://mlsanity.com/paper/2102.03866/) (2021)
- [Directly Attention Loss Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2311.14390/) (2023)
- [Improving Experience Replay with Successor Representation](https://mlsanity.com/paper/2111.14331/) (2021)
- [MEET: A Monte Carlo Exploration-Exploitation Trade-off for Buffer Sampling](https://mlsanity.com/paper/2210.13545/) (2022)
- [EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2501.09611/) (2025)
- [Uncertainty-Driven Replay Memory for Reinforcement Learning](https://mlsanity.com/paper/2608.29860/) (2026)

## Abstract (arXiv)

Experience replay is widely used in deep reinforcement learning algorithms and allows agents to remember and learn from experiences from the past. In an effort to learn more efficiently, researchers proposed prioritized experience replay (PER) which samples important transitions more frequently. In this paper, we propose Prioritized Sequence Experience Replay (PSER) a framework for prioritizing sequences of experience in an attempt to both learn more efficiently and to obtain better performance. We compare the performance of PER and PSER sampling techniques in a tabular Q-learning environment and in DQN on the Atari 2600 benchmark. We prove theoretically that PSER is guaranteed to converge faster than PER and empirically show PSER substantially improves upon PER.