Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models
Not on the map. Why?
Abstract (arXiv)
Generative AI is gaining increasing attention in software engineering, where testing remains an indispensable reliability mechanism. According to the widely adopted testing pyramid, unit tests constitute the majority of test cases and are often schematic, requiring minimal domain expertise. Automatically generating such tests under the supervision of software engineers can significantly enhance productivity during the development phase of the software lifecycle. This paper investigates the impact of code context and prompting strategies on the quality and adequacy of unit tests generated by various large language models (LLMs) across several families. The results show that including docstrings notably improves code adequacy, while further extending context to the full implementation yields definitely smaller gains. Notably, the chain-of-thought prompting strategy -- applied even to 'reasoning' models -- achieves the best results, with up to 96.3% branch coverage, a 57% average mutation score, and near-perfect compilation success rate. Among the evaluated models, M5 (Gemini 2.5 Pro) demonstrated superior performance in both mutation score and branch coverage being still in top in terms of compilation success rate. All the code and resulting test suites are publicly available at https://github.com/peetery/LLM-analysis.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 46 references that are in the map. First posted on arXiv 18 Jul 2025.
Builds on
19 of 46 references.
References · 19
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
- 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
- 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
7 more
- 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
- 2023Art and the science of generative AI: A deeper divecited by 95
- 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
- 2021Hybrid Intelligencecited by 55
- 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 50
- 2024TESTEVAL: Benchmarking Large Language Models for Test Case Generationcited by 40
- 2024A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suitescited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Artificial or Just Artful? Do LLMs Bend the Rules in Programming? ↗
- 2024Large-scale, Independent and Comprehensive study of the power of LLMs for test case generation
- 2024The Impact of Prompt Programming on Function-Level Code Generation
- 2026On the risk of coding before testing: An empirical study on LLM-based test generation workflow
- 2025From Code Generation to Software Testing: AI Copilot with Context-Based RAG
- 2024Understanding Defects in Generated Codes by Language Models
- 2025Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code
- 2025Dynamic Scaling of Unit Tests for Code Reward Modeling