# Can LLMs Generate User Stories and Assess Their Quality?

arXiv: https://arxiv.org/abs/2507.15157 · HTML: https://mlsanity.com/paper/2507.15157/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 71 references that are in the map.
- First posted on arXiv 20 Jul 2025.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [InferFix: End-to-End Program Repair with LLMs](https://mlsanity.com/paper/2303.07263/) (2023)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models](https://mlsanity.com/paper/2303.04864/) (2023)
- [Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair](https://mlsanity.com/paper/2309.00608/) (2023)
- [Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs](https://mlsanity.com/paper/2310.13976/) (2023)
- [Investigating ChatGPT's Potential to Assist in Requirements Elicitation Processes](https://mlsanity.com/paper/2307.07381/) (2023)
- [Requirements are All You Need: From Requirements to Code with LLMs](https://mlsanity.com/paper/2406.10101/) (2024)
- [Using LLMs in Software Requirements Specifications: An Empirical Evaluation](https://mlsanity.com/paper/2404.17842/) (2024)
- [Elicitron: An LLM Agent-Based Simulation Framework for Design Requirements Elicitation](https://mlsanity.com/paper/2404.16045/) (2024)
- [Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability](https://mlsanity.com/paper/2308.00229/) (2023)
- [Leveraging LLMs for the Quality Assurance of Software Requirements](https://mlsanity.com/paper/2408.10886/) (2024)
- [Engineering Safety Requirements for Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2403.16289/) (2024)
- [ContraDoc: Understanding Self-Contradictions in Documents with Large Language Models](https://mlsanity.com/paper/2311.09182/) (2023)
- [Studying LLM Performance on Closed- and Open-source Data](https://mlsanity.com/paper/2402.15100/) (2024)
- [Generative AI for Requirements Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2409.06741/) (2024)
- [Improving Requirements Completeness: Automated Assistance through Large Language Models](https://mlsanity.com/paper/2308.03784/) (2023)
- [Exploring LLMs for Verifying Technical System Specifications Against Requirements](https://mlsanity.com/paper/2411.11582/) (2024)
- [Requirements Elicitation Follow-Up Question Generation](https://mlsanity.com/paper/2507.02858/) (2025)
- [How Do Requirements Evolve During Elicitation? An Empirical Study Combining Interviews and App Store Analysis](https://mlsanity.com/paper/2208.00825/) (2022)
- [Strategies, Benefits and Challenges of App Store-inspired Requirements Elicitation](https://mlsanity.com/paper/2301.12090/) (2023)

## Built on by (most cited first, 2 of 2)

- [Automated Alignment between Elicitation Interviews and Requirements](https://mlsanity.com/paper/2510.08622/) (2025)
- [ReqGenX: An Empirical Study of Atomic Decomposition, Artifact Regeneration, and Reconstruction for Legacy SRS Documents](https://mlsanity.com/paper/2607.16564/) (2026)

## Abstract (arXiv)

Requirements elicitation is still one of the most challenging activities of the requirements engineering process due to the difficulty requirements analysts face in understanding and translating complex needs into concrete requirements. In addition, specifying high-quality requirements is crucial, as it can directly impact the quality of the software to be developed. Although automated tools allow for assessing the syntactic quality of requirements, evaluating semantic metrics (e.g., language clarity, internal consistency) remains a manual and time-consuming activity. This paper explores how LLMs can help automate requirements elicitation within agile frameworks, where requirements are defined as user stories (US). We used 10 state-of-the-art LLMs to investigate their ability to generate US automatically by emulating customer interviews. We evaluated the quality of US generated by LLMs, comparing it with the quality of US generated by humans (domain experts and students). We also explored whether and how LLMs can be used to automatically evaluate the semantic quality of US. Our results indicate that LLMs can generate US similar to humans in terms of coverage and stylistic quality, but exhibit lower diversity and creativity. Although LLM-generated US are generally comparable in quality to those created by humans, they tend to meet the acceptance quality criteria less frequently, regardless of the scale of the LLM model. Finally, LLMs can reliably assess the semantic quality of US when provided with clear evaluation criteria and have the potential to reduce human effort in large-scale assessments.