# Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools

arXiv: https://arxiv.org/abs/2312.10622 · HTML: https://mlsanity.com/paper/2312.10622/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 17 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation](https://mlsanity.com/paper/2305.04207/) (2023)
- [Code4Struct: Code Generation for Few-Shot Event Structure Prediction](https://mlsanity.com/paper/2210.12810/) (2022)
- [TOGA: A Neural Method for Test Oracle Generation](https://mlsanity.com/paper/2109.09262/) (2021)
- [Pynguin: Automated Unit Test Generation for Python](https://mlsanity.com/paper/2202.05218/) (2022)
- [Interactive Code Generation via Test-Driven User-Intent Formalization](https://mlsanity.com/paper/2208.05950/) (2022)
- [LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities](https://mlsanity.com/paper/2309.13574/) (2023)
- [An Empirical Study of Automated Unit Test Generation for Python](https://mlsanity.com/paper/2111.05003/) (2021)
- [Basic Block Coverage for Search-based Unit Testing and Crash Reproduction](https://mlsanity.com/paper/2203.02337/) (2022)

## Built on by (most cited first, 13 of 13)

- [The Future of AI-Driven Software Engineering](https://mlsanity.com/paper/2406.07737/) (2024)
- [A Contemporary Survey of Large Language Model Assisted Program Analysis](https://mlsanity.com/paper/2502.18474/) (2025)
- [PromptPex: Automatic Test Generation for Language Model Prompts](https://mlsanity.com/paper/2503.05070/) (2025)
- [Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropy](https://mlsanity.com/paper/2411.08254/) (2024)
- [Do Autonomous Agents Contribute Test Code? A Study of Tests in Agentic Pull Requests](https://mlsanity.com/paper/2601.03556/) (2026)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [QiMeng-MuPa: Mutual-Supervised Learning for Sequential-to-Parallel Code Translation](https://mlsanity.com/paper/2506.11153/) (2025)
- [DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Pattern](https://mlsanity.com/paper/2509.24975/) (2025)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [CUBETESTERAI: Automated JUnit Test Generation using the LLaMA Model](https://mlsanity.com/paper/2504.15286/) (2025)
- [ReCatcher: Towards LLMs Regression Testing for Code Generation](https://mlsanity.com/paper/2507.19390/) (2025)
- [Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)](https://mlsanity.com/paper/2607.22880/) (2026)
- [VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories](https://mlsanity.com/paper/2609.05978/) (2026)

## Abstract (arXiv)

Generating unit tests is a crucial task in software development, demanding substantial time and effort from programmers. The advent of Large Language Models (LLMs) introduces a novel avenue for unit test script generation. This research aims to experimentally investigate the effectiveness of LLMs, specifically exemplified by ChatGPT, for generating unit test scripts for Python programs, and how the generated test cases compare with those generated by an existing unit test generator (Pynguin). For experiments, we consider three types of code units: 1) Procedural scripts, 2) Function-based modular code, and 3) Class-based code. The generated test cases are evaluated based on criteria such as coverage, correctness, and readability. Our results show that ChatGPT's performance is comparable with Pynguin in terms of coverage, though for some cases its performance is superior to Pynguin. We also find that about a third of assertions generated by ChatGPT for some categories were incorrect. Our results also show that there is minimal overlap in missed statements between ChatGPT and Pynguin, thus, suggesting that a combination of both tools may enhance unit test generation performance. Finally, in our experiments, prompt engineering improved ChatGPT's performance, achieving a much higher coverage.