# Correcting Within-Group Self-Selection Bias in Prioritized Replay

arXiv: https://arxiv.org/abs/2609.25297 · HTML: https://mlsanity.com/paper/2609.25297/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction](https://mlsanity.com/paper/2003.07305/) (2020)
- [An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay](https://mlsanity.com/paper/2007.06049/) (2020)
- [Prioritizing Samples in Reinforcement Learning with Reducible Loss](https://mlsanity.com/paper/2208.10483/) (2022)
- [Understanding and Mitigating the Limitations of Prioritized Experience Replay](https://mlsanity.com/paper/2007.09569/) (2020)
- [Large Batch Experience Replay](https://mlsanity.com/paper/2110.01528/) (2021)
- [Revisiting Prioritized Experience Replay: A Value Perspective](https://mlsanity.com/paper/2102.03261/) (2021)
- [Uncertainty Prioritized Experience Replay](https://mlsanity.com/paper/2506.09270/) (2025)
- [ROER: Regularized Optimal Experience Replay](https://mlsanity.com/paper/2407.03995/) (2024)

## Abstract (arXiv)

Prioritized experience replay (PER) improves sample efficiency by replaying high-priority transitions, usually according to absolute temporal-difference error. In stochastic environments, PER can distort the distribution of realized outcomes replayed from transitions with the same state-action pair. We call this within-group self-selection. We quantify the resulting changes in within-group outcome frequencies and mean Bellman targets. We decompose PER into between-group allocation and conditional sibling selection, and derive fixed-buffer corrections that preserve current group-level priority mass: SAMPLE selects a group through PER and trains on a uniformly sampled sibling; AVG averages sibling Bellman targets; and MODEL samples from an empirical full-outcome model. In exact state-action environments with rare high-magnitude outcomes, sibling-aware replay improves learning efficiency over PER, although matched parameter sweeps show that tuning can narrow some gaps. In MinAtar, approximate VQ-VAE groups with SAMPLE mitigate degradation under mean-preserving reward tails in four of five games. Sibling-aware replay thus retains the focus on high-priority state-action regions while recovering their empirical outcome frequencies.