# Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns

arXiv: https://arxiv.org/abs/2607.00048 · HTML: https://mlsanity.com/paper/2607.00048/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 9 of 9)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [LLM-based agents for automating the enhancement of user story quality: An early report](https://mlsanity.com/paper/2403.09442/) (2024)
- [Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code](https://mlsanity.com/paper/2310.10508/) (2023)
- [Exploring Human-AI Collaboration in Agile: Customised LLM Meeting Assistants](https://mlsanity.com/paper/2404.14871/) (2024)
- [Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks](https://mlsanity.com/paper/2506.05614/) (2025)
- [Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams](https://mlsanity.com/paper/2311.14169/) (2023)
- [Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Exam](https://mlsanity.com/paper/2406.09671/) (2024)
- [Evaluating Large Language Models on the GMAT: Implications for the Future of Business Education](https://mlsanity.com/paper/2401.02985/) (2024)
- [Assessing the Capability of LLMs in Solving POSCOMP Questions](https://mlsanity.com/paper/2505.20338/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed. In Software Engineering, such settings are particularly relevant when questions depend on strict adherence to normative definitions, roles, artifacts, and rules. This paper evaluates the performance of three contemporary LLMs, textitGPT-5 mini, textitGemini 3 Flash, and textitDeepSeek Chat 3.2, in answering 993 Scrum certification-style questions aligned with the Professional Scrum Master I (PSM I) assessment format. We evaluated the models under three prompting strategies (textitzero-shot, textitchain-of-thought, and textitsource-grounded), with repeated executions to assess intra-model stability. We also analyzed performance across Scrum topics and question formats, complemented by a qualitative analysis of recurring error patterns in incorrect answers. Results revealed clear differences among models, with Gemini 3 Flash achieving the highest accuracy, followed by GPT-5 mini and DeepSeek Chat 3.2, while intra-model variability remained low across all conditions. By question format, the models achieved the highest accuracy on single-answer multiple-choice items, whereas multi-select and True/False questions were more error-prone. By topic, performance was more consistent in normatively explicit areas such as Artifacts, Empiricism, and Product Value, but more fragile in Scrum Values, Self-Managing Teams, and Stakeholders & Customers. The qualitative analysis showed that errors were systematic rather than random, involving overgeneralization, restrictive wording, compound distractors, and conflicts between common market interpretations and strict Scrum definitions.