mlsanityData to 7 Oct 2026

Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiences

arXiv 2405.14629 ↗First posted 23 May 2024cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning (RL) with experience replay, experiences stored in a replay buffer influence the RL agent's performance. Information about how these experiences influence the agent's performance is valuable for various purposes, such as identifying experiences that negatively influence underperforming agents. One method for estimating the influence of experiences is the leave-one-out (LOO) method. However, this method is usually computationally prohibitive. In this paper, we present Policy Iteration with Turn-over Dropout (PIToD), which efficiently estimates the influence of experiences. We evaluate how correctly PIToD estimates the influence of experiences and its efficiency compared to LOO. We then apply PIToD to amend underperforming RL agents, i.e., we use PIToD to estimate negatively influential experiences for the RL agents and to delete the influence of these experiences. We show that RL agents' performance is significantly improved via amendments with PIToD.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 57 of its 77 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

57 of 77 references.

References · 57

  1. 2016Layer Normalizationcited by 5,579
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2015Prioritized Experience Replaycited by 1,070
38 more
  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  3. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  5. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  6. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  7. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  8. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  9. 2020Revisiting Fundamentals of Experience Replaycited by 171
  10. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  11. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 146
  12. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  13. 2017Visualizing and Understanding Atari Agentscited by 141
  14. 2021Scaling Up Influence Functionscited by 130
  15. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  16. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  17. 2017A Deeper Look at Experience Replaycited by 104
  18. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  19. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 77
  20. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  21. 2016Graying the black box: Understanding DQNscited by 68
  22. 2019Data Cleansing for Models Trained with SGDcited by 67
  23. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  24. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  25. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  26. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  27. 2018PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Makingcited by 51
  28. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  29. 2016Towards Deep Symbolic Reinforcement Learningcited by 42
  30. 2018Remember and Forget for Experience Replaycited by 42
  31. 2020Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitionscited by 41
  32. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  33. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  34. 2023BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learningcited by 29
  35. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  36. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 25
  37. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasetscited by 24
  38. 2017The Effects of Memory Replay in Reinforcement Learningcited by 22

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Data Attribution in Adaptive Learningcited by 0