# What Do Users Ask in Open-Source AI Repositories? An Empirical Study of GitHub Issues

arXiv: https://arxiv.org/abs/2303.09795 · HTML: https://mlsanity.com/paper/2303.09795/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2023.

## Built on by (most cited first, 8 of 8)

- [PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software](https://mlsanity.com/paper/2402.00699/) (2024)
- [AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection](https://mlsanity.com/paper/2608.06790/) (2026)
- [REAgent: Requirement-Driven LLM Agents for Software Issue Resolution](https://mlsanity.com/paper/2604.06861/) (2026)
- [Demystifying Issues, Causes and Solutions in LLM Open-Source Projects](https://mlsanity.com/paper/2409.16559/) (2024)
- [PeaTMOSS: Mining Pre-Trained Models in Open-Source Software](https://mlsanity.com/paper/2310.03620/) (2023)
- [Securing the AI Supply Chain: What Can We Learn From Developer-Reported Security Issues and Solutions of AI Projects?](https://mlsanity.com/paper/2512.23385/) (2025)
- [A Taxonomy of Runtime Faults in Model Context Protocol Servers](https://mlsanity.com/paper/2606.05339/) (2026)
- [Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning Projects](https://mlsanity.com/paper/2402.16546/) (2024)

## Abstract (arXiv)

Artificial Intelligence systems, which benefit from the availability of large-scale datasets and increasing computational power, have become effective solutions to various critical tasks, such as natural language understanding, speech recognition, and image processing. The advancement of these AI systems is inseparable from open-source software (OSS). This paper presents an empirical study that investigates the issues in the repositories of open-source AI repositories to assist developers in understanding problems during the process of employing AI systems. We collect 576 repositories from the PapersWithCode platform. Among these repositories, we find 24,953 issues by utilizing GitHub REST APIs. Our empirical study includes three phases. First, we manually analyze these issues to categorize the problems that developers are likely to encounter in open-source AI repositories. Specifically, we provide a taxonomy of 13 categories related to AI systems. The two most common issues are runtime errors (23.18%) and unclear instructions (19.53%). Second, we see that 67.5% of issues are closed. We also find that half of these issues resolve within four days. Moreover, issue management features, e.g., label and assign, are not widely adopted in open-source AI repositories. In particular, only 7.81% and 5.9% of repositories label issues and assign these issues to assignees, respectively. Finally, we empirically show that employing GitHub issue management features and writing issues with detailed descriptions facilitate the resolution of issues. Based on our findings, we make recommendations for developers to help better manage the issues of open-source AI repositories and improve their quality.