mlsanityData to 14 Oct 2026

Using Large Language Models to Generate JUnit Tests: An Empirical Study

arXiv 2305.00418 ↗First posted 30 Apr 2023cs.SE cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well three models (Codex, GPT-3.5-Turbo, and StarCoder) can generate unit tests. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the effect of context generation on the unit test generation process. We evaluated the models based on compilation rates, test correctness, test coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 17 of its 74 references that are in the map. First posted on arXiv 30 Apr 2023.

Builds on

17 of 74 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  4. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  5. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  6. 2022CodeT: Code Generation with Generated Testscited by 384
  7. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 221
  8. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  9. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  10. 2022Productivity Assessment of Neural Code Completioncited by 98
  11. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  12. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 50
5 more
  1. 2020Fast and Memory-Efficient Neural Code Completioncited by 38
  2. 2023Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Promptingcited by 23
  3. 2022M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarizationcited by 14
  4. 2022On the Effectiveness of Pretrained Models for API Learningcited by 9
  5. 2017A Large-Scale Study on the Usage of Testing Patterns that Address Maintainability Attributes (Patterns for Ease of Modification, Diagnoses, and Comprehension)cited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 14; 2025: 23; 2026: 12.

53 papers.

Sort:

By area:

  1. 2024The Limits of Inference Scaling Through Resamplingcited by 53
  2. 2023SALLM: Security Assessment of Generated Codecited by 39
  3. 2024The Future of AI-Driven Software Engineeringcited by 23
  4. 2023FRANC: A Lightweight Framework for High-Quality Code Generationcited by 14
  5. 2025CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generationcited by 14
  6. 2025Migrating Code At Scale With LLMs At Googlecited by 14
  7. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  8. 2024Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Testscited by 11
  9. 2024The Fault in our Stars: Quality Assessment of Code Generation Benchmarkscited by 9
  10. 2025Do LLMs Consider Security? An Empirical Study on Responses to Programming Questionscited by 9
  11. 2025PromptPex: Automatic Test Generation for Language Model Promptscited by 9
  12. 2025LLM Test Generation via Iterative Hybrid Program Analysiscited by 9
38 more
  1. 2025Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generationcited by 9
  2. 2024A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suitescited by 8
  3. 2025Benchmarking LLMs for Unit Test Generation from Real-World Functionscited by 8
  4. 2026Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generationcited by 7
  5. 2024Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programmingcited by 6
  6. 2025How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-benchcited by 5
  7. 2024exLong: Generating Exceptional Behavior Tests with Large Language Modelscited by 4
  8. 2025MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanismscited by 4
  9. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  10. 2025Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesiscited by 4
  11. 2023Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Coursescited by 3
  12. 2024Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Studycited by 3
  13. 2025SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executorscited by 3
  14. 2025On Simulation-Guided LLM-based Code Generation for Safe Autonomous Driving Softwarecited by 3
  15. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  16. 2024Forklift: An Extensible Neural Liftercited by 2
  17. 2024Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context Learningcited by 2
  18. 2024Evaluation of the Programming Skills of Large Language Modelscited by 2
  19. 2024Exploring the Integration of Large Language Models in Industrial Test Maintenance Processescited by 2
  20. 2025An Analysis of LLM Fine-Tuning and Few-Shot Learning for Flaky Test Detection and Classificationcited by 2
  21. 2025QiMeng-MuPa: Mutual-Supervised Learning for Sequential-to-Parallel Code Translationcited by 2
  22. 2025Large Language Models for Software Testing: A Research Roadmapcited by 2
  23. 2024On the Effectiveness of LLMs for Manual Test Verificationscited by 1
  24. 2025A Systematic Approach for Assessing Large Language Models' Test Case Generation Capabilitycited by 1
  25. 2025Large Language Models in Code Co-generation for Safe Autonomous Vehiclescited by 1
  26. 2025LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Frameworkcited by 1
  27. 2026Configuration Smells in AGENTS.md Files: Common Mistakes in Configuring Coding Agentscited by 1
  28. 2026MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Modelscited by 1
  29. 2026SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generationcited by 1
  30. 2026Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learningcited by 1
  31. 2023Breaking the Silence: the Threats of Using LLMs in Software Engineeringcited by 0
  32. 2025CUBETESTERAI: Automated JUnit Test Generation using the LLaMA Modelcited by 0
  33. 2025Evaluating Large Language Models for the Generation of Unit Tests with Equivalence Partitions and Boundary Valuescited by 0
  34. 2025iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocolscited by 0
  35. 2026Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automationcited by 0
  36. 2026Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semanticscited by 0
  37. 2026Fail-Aware and Explainable Test Oracle Predictioncited by 0
  38. 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requestscited by 0

50 most cited of 53 All 53 (JSON)