# A Case Study Investigating the Role of Generative AI in Quality Evaluations of Epics in Agile Software Development

arXiv: https://arxiv.org/abs/2505.07664 · HTML: https://mlsanity.com/paper/2505.07664/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 78 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 19 of 19)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation](https://mlsanity.com/paper/2402.14978/) (2024)
- [A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration](https://mlsanity.com/paper/2404.00405/) (2024)
- [Human-Centered Design Recommendations for LLM-as-a-Judge](https://mlsanity.com/paper/2407.03479/) (2024)
- [Investigating ChatGPT's Potential to Assist in Requirements Elicitation Processes](https://mlsanity.com/paper/2307.07381/) (2023)
- [Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course](https://mlsanity.com/paper/2407.05216/) (2024)
- [Black-box Uncertainty Quantification Method for LLM-as-a-Judge](https://mlsanity.com/paper/2410.11594/) (2024)
- [Leveraging LLMs for the Quality Assurance of Software Requirements](https://mlsanity.com/paper/2408.10886/) (2024)
- [ChatCoder: Chat-based Refine Requirement Improves LLMs' Code Generation](https://mlsanity.com/paper/2311.00272/) (2023)
- [AI based Multiagent Approach for Requirements Elicitation and Analysis](https://mlsanity.com/paper/2409.00038/) (2024)
- [LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization](https://mlsanity.com/paper/2409.00630/) (2024)
- [Generative AI Uses and Risks for Knowledge Workers in a Science Organization](https://mlsanity.com/paper/2501.16577/) (2025)
- [A Survey on Large Language Models from Concept to Implementation](https://mlsanity.com/paper/2403.18969/) (2024)
- [Documentation of quality requirements in agile software development](https://mlsanity.com/paper/2010.02971/) (2020)

## Built on by (most cited first, 1 of 1)

- [The State of Generative AI in Software Development: Insights from Literature and a Developer Survey](https://mlsanity.com/paper/2603.16975/) (2026)

## Abstract (arXiv)

The broad availability of generative AI offers new opportunities to support various work domains, including agile software development. Agile epics are a key artifact for product managers to communicate requirements to stakeholders. However, in practice, they are often poorly defined, leading to churn, delivery delays, and cost overruns. In this industry case study, we investigate opportunities for large language models (LLMs) to evaluate agile epic quality in a global company. Results from a user study with 17 product managers indicate how LLM evaluations could be integrated into their work practices, including perceived values and usage in improving their epics. High levels of satisfaction indicate that agile epics are a new, viable application of AI evaluations. However, our findings also outline challenges, limitations, and adoption barriers that can inform both practitioners and researchers on the integration of such evaluations into future agile work practices.