# Applying Large Language Models to Issue Classification: Revisiting with Extended Data and New Models

arXiv: https://arxiv.org/abs/2506.00128 · HTML: https://mlsanity.com/paper/2506.00128/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Multi-task Learning based Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2012.14631/) (2020)
- [Towards Human-Bot Collaborative Software Architecting with ChatGPT](https://mlsanity.com/paper/2302.14600/) (2023)
- [Topic Recommendation for Software Repositories using Multi-label Classification Algorithms](https://mlsanity.com/paper/2010.09116/) (2020)
- [Traceability Transformed: Generating more Accurate Links with Pre-Trained BERT Models](https://mlsanity.com/paper/2102.04411/) (2021)
- [Predicting the Objective and Priority of Issue Reports in Software Repositories](https://mlsanity.com/paper/2012.10951/) (2020)
- [Predicting Issue Types on GitHub](https://mlsanity.com/paper/2107.09936/) (2021)
- [Can I Solve It? Identifying APIs Required to Complete OSS Task](https://mlsanity.com/paper/2103.12653/) (2021)
- [A Simple NLP-based Approach to Support Onboarding and Retention in Open Source Communities](https://mlsanity.com/paper/1806.02592/) (2018)
- [A Bug or a Suggestion? An Automatic Way to Label Issues](https://mlsanity.com/paper/1909.00934/) (2019)
- [An Empirical Comparison of Developer Retention in the RubyGems and npm Software Ecosystems](https://mlsanity.com/paper/1708.02618/) (2017)
- [Bug or not bug? That is the question](https://mlsanity.com/paper/2103.12218/) (2021)
- [GiveMeLabeledIssues: An Open Source Issue Recommendation System](https://mlsanity.com/paper/2303.13418/) (2023)
- [Semantically-enhanced Topic Recommendation System for Software Projects](https://mlsanity.com/paper/2206.00085/) (2022)
- [Tag that issue: Applying API-domain labels in issue tracking systems](https://mlsanity.com/paper/2304.02877/) (2023)
- [SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub](https://mlsanity.com/paper/2501.15922/) (2025)
- [SPRINT: An Assistant for Issue Report Management](https://mlsanity.com/paper/2502.04147/) (2025)

## Built on by (most cited first, 1 of 1)

- [LIA: Supervised Fine-Tuning of Large Language Models for Automatic Issue Assignment](https://mlsanity.com/paper/2601.01780/) (2026)

## Abstract (arXiv)

Effective prioritization of issue reports in software engineering helps to optimize resource allocation and information recovery. However, manual issue classification is laborious and lacks scalability. As an alternative, many open source software (OSS) projects employ automated processes for this task, yet this method often relies on large datasets for adequate training. Traditionally, machine learning techniques have been used for issue classification. More recently, large language models (LLMs) have emerged as powerful tools for addressing a range of software engineering challenges, including code and test generation, mapping new requirements to legacy software endpoints, and conducting code reviews. The following research investigates an automated approach to issue classification based on LLMs. By leveraging the capabilities of such models, we aim to develop a robust system for prioritizing issue reports, mitigating the necessity for extensive training data while also maintaining reliability in classification. In our research, we developed an LLM-based approach for accurately labeling issues by selecting two of the most prominent large language models. We then compared their performance across multiple datasets. Our findings show that GPT-4o achieved the best results in classifying issues from the NLBSE 2024 competition. Moreover, GPT-4o outperformed DeepSeek R1, achieving an F1 score 20% higher when both models were trained on the same dataset from the NLBSE 2023 competition, which was ten times larger than the NLBSE 2024 dataset. The fine-tuned GPT-4o model attained an average F1 score of 80.7%, while the fine-tuned DeepSeek R1 model achieved 59.33%. Increasing the dataset size did not improve the F1 score, reducing the dependence on massive datasets for building an efficient solution to issue classification.