mlsanityData to 7 Oct 2026

Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools

arXiv 2312.10622 ↗First posted 17 Dec 2023cs.SE cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Generating unit tests is a crucial task in software development, demanding substantial time and effort from programmers. The advent of Large Language Models (LLMs) introduces a novel avenue for unit test script generation. This research aims to experimentally investigate the effectiveness of LLMs, specifically exemplified by ChatGPT, for generating unit test scripts for Python programs, and how the generated test cases compare with those generated by an existing unit test generator (Pynguin). For experiments, we consider three types of code units: 1) Procedural scripts, 2) Function-based modular code, and 3) Class-based code. The generated test cases are evaluated based on criteria such as coverage, correctness, and readability. Our results show that ChatGPT's performance is comparable with Pynguin in terms of coverage, though for some cases its performance is superior to Pynguin. We also find that about a third of assertions generated by ChatGPT for some categories were incorrect. Our results also show that there is minimal overlap in missed statements between ChatGPT and Pynguin, thus, suggesting that a combination of both tools may enhance unit test generation performance. Finally, in our experiments, prompt engineering improved ChatGPT's performance, achieving a much higher coverage.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 14 of its 37 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

14 of 37 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  6. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  7. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  8. 2022Code4Struct: Code Generation for Few-Shot Event Structure Predictioncited by 70
  9. 2021TOGA: A Neural Method for Test Oracle Generationcited by 57
  10. 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
  11. 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
  12. 2023LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunitiescited by 19
2 more
  1. 2021An Empirical Study of Automated Unit Test Generation for Pythoncited by 10
  2. 2022Basic Block Coverage for Search-based Unit Testing and Crash Reproductioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024The Future of AI-Driven Software Engineeringcited by 23
  2. 2025A Contemporary Survey of Large Language Model Assisted Program Analysiscited by 10
  3. 2025PromptPex: Automatic Test Generation for Language Model Promptscited by 9
  4. 2026Do Autonomous Agents Contribute Test Code? A Study of Tests in Agentic Pull Requestscited by 4
  5. 2024Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropycited by 3
  6. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  7. 2025QiMeng-MuPa: Mutual-Supervised Learning for Sequential-to-Parallel Code Translationcited by 2
  8. 2025DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Patterncited by 2
  9. 2025Large Language Models for Software Testing: A Research Roadmapcited by 2
  10. 2025CUBETESTERAI: Automated JUnit Test Generation using the LLaMA Modelcited by 0
  11. 2025ReCatcher: Towards LLMs Regression Testing for Code Generationcited by 0
  12. 2026Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)cited by 0
1 more
  1. 2026VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositoriescited by 0