# LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework

arXiv: https://arxiv.org/abs/2511.20403 · HTML: https://mlsanity.com/paper/2511.20403/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020)
- [Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models](https://mlsanity.com/paper/2212.14834/) (2022)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation](https://mlsanity.com/paper/2305.04207/) (2023)
- [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024)
- [Using Large Language Models to Generate JUnit Tests: An Empirical Study](https://mlsanity.com/paper/2305.00418/) (2023)
- [TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark](https://mlsanity.com/paper/2410.00752/) (2024)
- [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://mlsanity.com/paper/2308.16557/) (2023)
- [A3Test: Assertion-Augmented Automated Test Case Generation](https://mlsanity.com/paper/2302.10352/) (2023)
- [ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation](https://mlsanity.com/paper/2307.00588/) (2023)
- [Learning Deep Semantics for Test Completion](https://mlsanity.com/paper/2302.10166/) (2023)
- [TestBench: Evaluating Class-Level Test Case Generation Capability of Large Language Models](https://mlsanity.com/paper/2409.17561/) (2024)
- [WebApp1K: A Practical Code-Generation Benchmark for Web App Development](https://mlsanity.com/paper/2408.00019/) (2024)
- [Methods2Test: A dataset of focal methods mapped to test cases](https://mlsanity.com/paper/2203.12776/) (2022)
- [TestSpark: IntelliJ IDEA's Ultimate Test Generation Companion](https://mlsanity.com/paper/2401.06580/) (2024)
- [Basic Block Coverage for Search-based Unit Testing and Crash Reproduction](https://mlsanity.com/paper/2203.02337/) (2022)
- [MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms](https://mlsanity.com/paper/2502.06556/) (2025)
- [JUGE: An Infrastructure for Benchmarking Java Unit Test Generators](https://mlsanity.com/paper/2106.07520/) (2021)

## Built on by (most cited first, 1 of 1)

- [TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution](https://mlsanity.com/paper/2607.02469/) (2026)

## Abstract (arXiv)

Unit testing is an essential but resource-intensive step in software development, ensuring individual code units function correctly. This paper introduces AgoneTest, an automated evaluation framework for Large Language Model-generated (LLM) unit tests in Java. AgoneTest does not aim to propose a novel test generation algorithm; rather, it supports researchers and developers in comparing different LLMs and prompting strategies through a standardized end-to-end evaluation pipeline under realistic conditions. We introduce the Classes2Test dataset, which maps Java classes under test to their corresponding test classes, and a framework that integrates advanced evaluation metrics, such as mutation score and test smells, for a comprehensive assessment. Experimental results show that, for the subset of tests that compile, LLM-generated tests can match or exceed human-written tests in terms of coverage and defect detection. Our findings also demonstrate that enhanced prompting strategies contribute to test quality. AgoneTest clarifies the potential of LLMs in software testing and offers insights for future improvements in model design, prompt engineering, and testing practices.