# Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approach

arXiv: https://arxiv.org/abs/2101.00135 · HTML: https://mlsanity.com/paper/2101.00135/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 49 references that are in the map.
- First posted on arXiv 1 Jan 2021.

## Builds on (most cited first, 16 of 16)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker](https://mlsanity.com/paper/1701.01724/) (2017)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [An Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/1811.12560/) (2018)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [Learning to Fly by Crashing](https://mlsanity.com/paper/1704.05588/) (2017)
- [Repairing Deep Neural Networks: Fix Patterns and Challenges](https://mlsanity.com/paper/2005.00972/) (2020)
- [Understanding the Factors that Impact the Popularity of GitHub Repositories](https://mlsanity.com/paper/1606.04984/) (2016)

## Built on by (most cited first, 18 of 18)

- [Bugs in Large Language Models Generated Code: An Empirical Study](https://mlsanity.com/paper/2403.08937/) (2024)
- [Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow](https://mlsanity.com/paper/2112.13314/) (2021)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities](https://mlsanity.com/paper/2312.06056/) (2023)
- [Bugs in Machine Learning-based Systems: A Faultload Benchmark](https://mlsanity.com/paper/2206.12311/) (2022)
- [A Search-Based Testing Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2206.07813/) (2022)
- [Bug Characterization in Machine Learning-based Systems](https://mlsanity.com/paper/2307.14512/) (2023)
- [A Survey on Failure Analysis and Fault Injection in AI Systems](https://mlsanity.com/paper/2407.00125/) (2024)
- [Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2506.18245/) (2025)
- [Mutation Testing of Deep Reinforcement Learning Based on Real Faults](https://mlsanity.com/paper/2301.05651/) (2023)
- [Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challenges](https://mlsanity.com/paper/2308.12438/) (2023)
- [Toward Debugging Deep Reinforcement Learning Programs with RLExplorer](https://mlsanity.com/paper/2410.04322/) (2024)
- [Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networks](https://mlsanity.com/paper/2508.04925/) (2025)
- [A Taxonomy of Runtime Faults in Model Context Protocol Servers](https://mlsanity.com/paper/2606.05339/) (2026)
- [Fuzzy Logic Guided Reward Function Variation: An Oracle for Testing Reinforcement Learning Programs](https://mlsanity.com/paper/2406.19812/) (2024)
- [Applications and Challenges of Fairness APIs in Machine Learning Software](https://mlsanity.com/paper/2508.16377/) (2025)
- [A Comprehensive Study of Implementation Bugs in Multi-modal Agents](https://mlsanity.com/paper/2607.04974/) (2026)
- [Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents](https://mlsanity.com/paper/2608.17524/) (2026)

## Abstract (arXiv)

A growing demand is witnessed in both industry and academia for employing Deep Learning (DL) in various domains to solve real-world problems. Deep Reinforcement Learning (DRL) is the application of DL in the domain of Reinforcement Learning (RL). Like any software systems, DRL applications can fail because of faults in their programs. In this paper, we present the first attempt to categorize faults occurring in DRL programs. We manually analyzed 761 artifacts of DRL programs (from Stack Overflow posts and GitHub issues) developed using well-known DRL frameworks (OpenAI Gym, Dopamine, Keras-rl, Tensorforce) and identified faults reported by developers/users. We labeled and taxonomized the identified faults through several rounds of discussions. The resulting taxonomy is validated using an online survey with 19 developers/researchers. To allow for the automatic detection of faults in DRL programs, we have defined a meta-model of DRL programs and developed DRLinter, a model-based fault detection approach that leverages static analysis and graph transformations. The execution flow of DRLinter consists in parsing a DRL program to generate a model conforming to our meta-model and applying detection rules on the model to identify faults occurrences. The effectiveness of DRLinter is evaluated using 15 synthetic DRLprograms in which we injected faults observed in the analyzed artifacts of the taxonomy. The results show that DRLinter can successfully detect faults in all synthetic faulty programs.