# Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent

arXiv: https://arxiv.org/abs/2512.14990 · HTML: https://mlsanity.com/paper/2512.14990/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 33 of its 82 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [Self-planning Code Generation with Large Language Models](https://mlsanity.com/paper/2303.06689/) (2023)
- [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [Compilable Neural Code Generation with Compiler Feedback](https://mlsanity.com/paper/2203.05132/) (2022)
- [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024)
- [Prompting Is All You Need: Automated Android Bug Replay with Large Language Models](https://mlsanity.com/paper/2306.01987/) (2023)
- [DeepLocalize: Fault Localization for Deep Neural Networks](https://mlsanity.com/paper/2103.03376/) (2021)
- [What do pre-trained code models know about code?](https://mlsanity.com/paper/2108.11308/) (2021)
- [Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding](https://mlsanity.com/paper/2112.02268/) (2021)
- [Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow](https://mlsanity.com/paper/2112.13314/) (2021)
- [An Empirical Study on the Usage of BERT Models for Code Completion](https://mlsanity.com/paper/2103.07115/) (2021)
- [DEAR: A Novel Deep Learning-based Approach for Automated Program Repair](https://mlsanity.com/paper/2205.01859/) (2022)
- [DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs](https://mlsanity.com/paper/2205.01938/) (2022)
- [Bugs in Machine Learning-based Systems: A Faultload Benchmark](https://mlsanity.com/paper/2206.12311/) (2022)
- [Bug Characterization in Machine Learning-based Systems](https://mlsanity.com/paper/2307.14512/) (2023)
- [Demystifying Dependency Bugs in Deep Learning Stack](https://mlsanity.com/paper/2207.10347/) (2022)
- [RLocator: Reinforcement Learning for Bug Localization](https://mlsanity.com/paper/2305.05586/) (2023)
- [Feedback-Driven Automated Whole Bug Report Reproduction for Android Apps](https://mlsanity.com/paper/2407.05165/) (2024)
- [Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study](https://mlsanity.com/paper/2401.03069/) (2024)
- [Towards Understanding the Impact of Data Bugs on Deep Learning Models in Software Engineering](https://mlsanity.com/paper/2411.12137/) (2024)
- [Evolution of statistical analysis in empirical software engineering research: Current state and steps forward](https://mlsanity.com/paper/1706.00933/) (2017)
- [On Using GUI Interaction Data to Improve Text Retrieval-based Bug Localization](https://mlsanity.com/paper/2310.08083/) (2023)

## Built on by (most cited first, 1 of 1)

- [Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering](https://mlsanity.com/paper/2604.01437/) (2026)

## Abstract (arXiv)

Despite their wide adoption in various domains (e.g., healthcare, finance, software engineering), Deep Learning (DL)-based applications suffer from many bugs, failures, and vulnerabilities. Reproducing these bugs is essential for their resolution, but it is extremely challenging due to the inherent nondeterminism of DL models and their tight coupling with hardware and software environments. According to recent studies, only about 3% of DL bugs can be reliably reproduced using manual approaches. To address these challenges, we present RepGen, a novel, automated, and intelligent approach for reproducing deep learning bugs. RepGen constructs a learning-enhanced context from a project, develops a comprehensive plan for bug reproduction, employs an iterative generate-validate-refine mechanism, and thus generates such code using an LLM that reproduces the bug at hand. We evaluate RepGen on 106 real-world deep learning bugs and achieve a reproduction rate of 80.19%, a 19.81% improvement over the state-of-the-art measure. A developer study involving 27 participants shows that RepGen improves the success rate of DL bug reproduction by 23.35%, reduces the time to reproduce by 56.8%, and lowers participants' cognitive load.