# ChatGPT as a tool for User Story Quality Evaluation: Trustworthy Out of the Box?

arXiv: https://arxiv.org/abs/2306.12132 · HTML: https://mlsanity.com/paper/2306.12132/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2023.

## Built on by (most cited first, 13 of 13)

- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [Prompt Smells: An Omen for Undesirable Generative AI Outputs](https://mlsanity.com/paper/2401.12611/) (2024)
- [Evaluating and Enhancing Trustworthiness of LLMs in Perception Tasks](https://mlsanity.com/paper/2408.01433/) (2024)
- [LLMs Can Check Their Own Results to Mitigate Hallucinations in Traffic Understanding Tasks](https://mlsanity.com/paper/2409.12580/) (2024)
- [ReqInOne: A Large Language Model-Based Agent for Software Requirements Specification Generation](https://mlsanity.com/paper/2508.09648/) (2025)
- [Tapping in a Remote Vehicle's onboard LLM to Complement the Ego Vehicle's Field-of-View](https://mlsanity.com/paper/2408.10794/) (2024)
- [Goal2Story: A Multi-Agent Fleet based on Privately Enabled sLLMs for Impacting Mapping on Requirements Elicitation](https://mlsanity.com/paper/2503.13279/) (2025)
- [Multi-Agent LLMs as Ethics Advocates for AI-Based Systems](https://mlsanity.com/paper/2507.08392/) (2025)
- [Exploring the Capabilities of ChatGPT in Ancient Chinese Translation and Person Name Recognition](https://mlsanity.com/paper/2312.15304/) (2023)
- [Variability-Driven User-Story Generation using LLM and Triadic Concept Analysis](https://mlsanity.com/paper/2504.08666/) (2025)
- [Automated Alignment between Elicitation Interviews and Requirements](https://mlsanity.com/paper/2510.08622/) (2025)
- [Artificial Intelligence as a Catalyst for Innovation in Software Engineering](https://mlsanity.com/paper/2603.10994/) (2026)

## Abstract (arXiv)

In Agile software development, user stories play a vital role in capturing and conveying end-user needs, prioritizing features, and facilitating communication and collaboration within development teams. However, automated methods for evaluating user stories require training in NLP tools and can be time-consuming to develop and integrate. This study explores using ChatGPT for user story quality evaluation and compares its performance with an existing benchmark. Our study shows that ChatGPT's evaluation aligns well with human evaluation, and we propose a ``best of three'' strategy to improve its output stability. We also discuss the concept of trustworthiness in AI and its implications for non-experts using ChatGPT's unprocessed outputs. Our research contributes to understanding the reliability and applicability of AI in user story evaluation and offers recommendations for future research.