mlsanityData to 14 Oct 2026

Toward Debugging Deep Reinforcement Learning Programs with RLExplorer

arXiv 2410.04322 ↗First posted 6 Oct 2024cs.SE cs.AI cs.LGcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Deep reinforcement learning (DRL) has shown success in diverse domains such as robotics, computer games, and recommendation systems. However, like any other software system, DRL-based software systems are susceptible to faults that pose unique challenges for debugging and diagnosing. These faults often result in unexpected behavior without explicit failures and error messages, making debugging difficult and time-consuming. Therefore, automating the monitoring and diagnosis of DRL systems is crucial to alleviate the burden on developers. In this paper, we propose RLExplorer, the first fault diagnosis approach for DRL-based software systems. RLExplorer automatically monitors training traces and runs diagnosis routines based on properties of the DRL learning dynamics to detect the occurrence of DRL-specific faults. It then logs the results of these diagnoses as warnings that cover theoretical concepts, recommended practices, and potential solutions to the identified faults. We conducted two sets of evaluations to assess RLExplorer. Our first evaluation of faulty DRL samples from Stack Overflow revealed that our approach can effectively diagnose real faults in 83% of the cases. Our second evaluation of RLExplorer with 15 DRL experts/developers showed that (1) RLExplorer could identify 3.6 times more defects than manual debugging and (2) RLExplorer is easily integrated into DRL applications.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 77 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

33 of 77 references.

References · 33

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2016OpenAI Gymcited by 1,736
  3. 2018Exploration by Random Network Distillationcited by 993
  4. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  5. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  6. 2017Deep Reinforcement Learning: An Overviewcited by 333
  7. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  8. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  9. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  10. 2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
  11. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
  12. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
21 more
  1. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  2. 2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
  3. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  4. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 40
  5. 2021DeepLocalize: Fault Localization for Deep Neural Networkscited by 38
  6. 2018Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networkscited by 37
  7. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  8. 2017Efficient Parallel Methods for Deep Reinforcement Learningcited by 32
  9. 2019DeepFault: Fault Localization for Deep Neural Networkscited by 30
  10. 2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 21
  11. 2021An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applicationscited by 20
  12. 2021DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programscited by 19
  13. 2021Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approachcited by 18
  14. 2020t-Soft Update of Target Network for Deep Reinforcement Learningcited by 17
  15. 2022DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programscited by 16
  16. 2022A Search-Based Testing Approach for Deep Reinforcement Learning Agentscited by 13
  17. 2018Object-sensitive Deep Reinforcement Learningcited by 10
  18. 2021Automatic Fault Detection for Deep Learning Programs Using Graph Transformationscited by 9
  19. 2021Convergence Guarantees for Deep Epsilon Greedy Policy Learningcited by 6
  20. 2020Interactive Visualization for Debugging RLcited by 5
  21. 2022Testing Feedforward Neural Networks Training Programscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Ghost Policies: A New Paradigm for Understanding and Learning from Failure in Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from the Test: Self-Referential Differential Testing for Deep RL Agents
  2. 2026MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Training
  3. 2024Analyzing Adversarial Inputs in Deep Reinforcement Learning
  4. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
  5. 2024ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policies
  6. 2024Verification-Guided Shielding for Deep Reinforcement Learning
  7. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  8. 2026FARL: A Fully Automated Pipeline of RL Interface and Solver Design