# Event Tables for Efficient Experience Replay

arXiv: https://arxiv.org/abs/2211.00576 · HTML: https://mlsanity.com/paper/2211.00576/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 23 of 23)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Deep Successor Reinforcement Learning](https://mlsanity.com/paper/1606.02396/) (2016)
- [Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning](https://mlsanity.com/paper/2008.07971/) (2020)
- [Experience Replay Optimization](https://mlsanity.com/paper/1906.08387/) (2019)
- [Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning](https://mlsanity.com/paper/2103.14666/) (2021)
- [Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update](https://mlsanity.com/paper/1805.12375/) (2018)
- [Reverb: A Framework For Experience Replay](https://mlsanity.com/paper/2102.04736/) (2021)
- [BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learning](https://mlsanity.com/paper/1806.06161/) (2018)
- [Topological Experience Replay](https://mlsanity.com/paper/2203.15845/) (2022)
- [Periodic Q-Learning](https://mlsanity.com/paper/2002.09795/) (2020)
- [A Note on Target Q-learning For Solving Finite MDPs with A Generative Oracle](https://mlsanity.com/paper/2203.11489/) (2022)
- [Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy](https://mlsanity.com/paper/2009.13736/) (2020)
- [Experience Replay Using Transition Sequences](https://mlsanity.com/paper/1705.10834/) (2017)

## Built on by (most cited first, 4 of 4)

- [ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control](https://mlsanity.com/paper/2410.05225/) (2024)
- [Two-Memory Reinforcement Learning](https://mlsanity.com/paper/2304.10098/) (2023)
- [Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards](https://mlsanity.com/paper/2311.00426/) (2023)
- [Coachable agents for interactive gameplay](https://mlsanity.com/paper/2607.00642/) (2026)

## Abstract (arXiv)

Experience replay (ER) is a crucial component of many deep reinforcement learning (RL) systems. However, uniform sampling from an ER buffer can lead to slow convergence and unstable asymptotic behaviors. This paper introduces Stratified Sampling from Event Tables (SSET), which partitions an ER buffer into Event Tables, each capturing important subsequences of optimal behavior. We prove a theoretical advantage over the traditional monolithic buffer approach and combine SSET with an existing prioritized sampling strategy to further improve learning speed and stability. Empirical results in challenging MiniGrid domains, benchmark RL environments, and a high-fidelity car racing simulator demonstrate the advantages and versatility of SSET over existing ER buffer sampling approaches.