mlsanityData to 7 Oct 2026

ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation

arXiv 2307.00588 ↗First posted 2 Jul 2023cs.SEcited by 32

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) have demonstrated exceptional success in a wide range of general domain tasks, such as question answering and following instructions. Moreover, LLMs have shown potential in various software engineering applications. In this study, we present a systematic comparison of test suites generated by the ChatGPT LLM and the state-of-the-art SBST tool EvoSuite. Our comparison is based on several critical factors, including correctness, readability, code coverage, and bug detection capability. By highlighting the strengths and weaknesses of LLMs (specifically ChatGPT) in generating unit test cases compared to EvoSuite, this work provides valuable insights into the performance of LLMs in solving software engineering problems. Overall, our findings underscore the potential of LLMs in software engineering and pave the way for further research in this area.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jul 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 10; 2026: 8.

32 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  3. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  4. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  5. 2024Exploring and Unleashing the Power of Large Language Models in Automated Code Translationcited by 57
  6. 2023WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Modelscited by 37
  7. 2024RepoMasterEval: Evaluating Code Completion via Real-World Repositoriescited by 10
  8. 2025PromptPex: Automatic Test Generation for Language Model Promptscited by 9
  9. 2024A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suitescited by 8
  10. 2025Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generationcited by 8
  11. 2025CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verificationcited by 7
  12. 2024Experimenting with Multi-Agent Software Development: Towards a Unified Platformcited by 6
20 more
  1. 2025Improving Deep Assertion Generation via Fine-Tuning Retrieval-Augmented Pre-trained Language Modelscited by 4
  2. 2024FuzzAug: Data Augmentation by Coverage-guided Fuzzing for Neural Test Generationcited by 3
  3. 2024Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropycited by 3
  4. 2025Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translationcited by 3
  5. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  6. 2025STAF: Leveraging LLMs for Automated Attack Tree-Based Security Test Generationcited by 2
  7. 2025Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projectscited by 1
  8. 2025Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Modelscited by 1
  9. 2025LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Frameworkcited by 1
  10. 2026MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysiscited by 1
  11. 2023Breaking the Silence: the Threats of Using LLMs in Software Engineeringcited by 0
  12. 2024Automated Creation of Source Code Variants of a Cryptographic Hash Function Implementation Using Generative Pre-Trained Transformer Modelscited by 0
  13. 2025CUBETESTERAI: Automated JUnit Test Generation using the LLaMA Modelcited by 0
  14. 2026IntentTester: Intent-Driven Multi-agent Framework for Cross-Library Test Migrationcited by 0
  15. 2026Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystemcited by 0
  16. 2026Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)cited by 0
  17. 2026Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Testscited by 0
  18. 2026VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositoriescited by 0
  19. 2026CIT-CAD: Constraint Intent Tree-based CAD Code Generation and Verificationcited by 0
  20. 2026On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Qualitycited by 0