mlsanityData to 14 Oct 2026

Introspective Experience Replay: Look Back When Surprised

arXiv 2206.03171 ↗First posted 7 Jun 2022cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning (RL), experience replay-based sampling techniques play a crucial role in promoting convergence by eliminating spurious correlations. However, widely used methods such as uniform experience replay (UER) and prioritized experience replay (PER) have been shown to have sub-optimal convergence and high seed sensitivity respectively. To address these issues, we propose a novel approach called IntrospectiveExperience Replay (IER) that selectively samples batches of data points prior to surprising events. Our method builds upon the theoretically sound reverse experience replay (RER) technique, which has been shown to reduce bias in the output of Q-learning-type algorithms with linear function approximation. However, this approach is not always practical or reliable when using neural function approximation. Through empirical evaluations, we demonstrate that IER with neural function approximation yields reliable and superior performance compared toUER, PER, and hindsight experience replay (HER) across most tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

27 of 49 references.

References · 27

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2016OpenAI Gymcited by 1,736
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2015Prioritized Experience Replaycited by 1,099
  9. 2017Deep Reinforcement Learning that Matterscited by 754
  10. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  11. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  12. 2019Mean estimation and regression under heavy-tailed distributions--a surveycited by 136
15 more
  1. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 121
  2. 2019RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completioncited by 58
  3. 2020Least Squares Regression with Markovian Data: Fundamental Limits and Algorithmscited by 56
  4. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  5. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  6. 2018Recall Traces: Backtracking Models for Efficient Reinforcement Learningcited by 23
  7. 2019Competitive Experience Replaycited by 23
  8. 2021Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPscited by 19
  9. 2021Streaming Linear System Identification with Reverse Experience Replaycited by 17
  10. 2022Topological Experience Replaycited by 17
  11. 2019Generative predecessor models for sample-efficient imitation learningcited by 14
  12. 2021Large Batch Experience Replaycited by 14
  13. 2018QUOTA: The Quantile Option Architecture for Reinforcement Learningcited by 13
  14. 2019Reverse Experience Replaycited by 8
  15. 2021Convergence Results For Q-Learning With Experience Replaycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 2.

3 papers.

  1. 2023Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimizationcited by 6
  2. 2026Multiscale replay: A robust algorithm for stochastic variational inequalities with a Markovian buffercited by 2
  3. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0