# Towards Refining Developer Questions using LLM-Based Named Entity Recognition for Developer Chatroom Conversations

arXiv: https://arxiv.org/abs/2503.00673 · HTML: https://mlsanity.com/paper/2503.00673/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 74 references that are in the map.
- First posted on arXiv 2 Mar 2025.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study](https://mlsanity.com/paper/2309.08221/) (2023)
- [Code and Named Entity Recognition in StackOverflow](https://mlsanity.com/paper/2005.01634/) (2020)
- [Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Study](https://mlsanity.com/paper/2402.16480/) (2024)
- [Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models](https://mlsanity.com/paper/2311.08921/) (2023)
- [LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking](https://mlsanity.com/paper/2404.05624/) (2024)
- [How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow](https://mlsanity.com/paper/1710.04692/) (2017)
- [Detecting Refactoring Commits in Machine Learning Python Projects: A Machine Learning-Based Approach](https://mlsanity.com/paper/2404.06572/) (2024)
- [llmNER: (Zero|Few)-Shot Named Entity Recognition, Exploiting the Power of Large Language Models](https://mlsanity.com/paper/2406.04528/) (2024)
- [An Empirical Study on LLM-based Classification of Requirements-related Provisions in Food-safety Regulations](https://mlsanity.com/paper/2501.14683/) (2025)
- [Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)](https://mlsanity.com/paper/2502.00627/) (2025)
- [A First Look at Developers' Live Chat on Gitter](https://mlsanity.com/paper/2107.05823/) (2021)

## Abstract (arXiv)

In software engineering chatrooms, communication is often hindered by imprecise questions that cannot be answered. Recognizing key entities can be essential for improving question clarity and facilitating better exchange. However, existing research using natural language processing techniques often overlooks these software-specific nuances. In this paper, we introduce Software-specific Named Entity Recognition, Intent Detection, and Resolution Classification (SENIR), a labeling approach that leverages a Large Language Model to annotate entities, intents, and resolution status in developer chatroom conversations. To offer quantitative guidance for improving question clarity and resolvability, we build a resolution prediction model that leverages SENIR's entity and intent labels along with additional predictive features. We evaluate SENIR on the DISCO dataset using a subset of annotated chatroom dialogues. SENIR achieves an 86% F-score for entity recognition, a 71% F-score for intent detection, and an 89% F-score for resolution status classification. Furthermore, our resolution prediction model, tested with various sampling strategies (random undersampling and oversampling with SMOTE) and evaluation methods (5-fold cross-validation, 10-fold cross-validation, and bootstrapping), demonstrates AUC values ranging from 0.7 to 0.8. Key factors influencing resolution include positive sentiment and entities such as Programming Language and User Variable across multiple intents, while diagnostic entities are more relevant in error-related questions. Moreover, resolution rates vary significantly by intent: questions about API Usage and API Change achieve higher resolution rates, whereas Discrepancy and Review have lower resolution rates. A Chi-Square analysis confirms the statistical significance of these differences.