# TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance

arXiv: https://arxiv.org/abs/2601.18241 · HTML: https://mlsanity.com/paper/2601.18241/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 30 of 31)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [Is Self-Repair a Silver Bullet for Code Generation?](https://mlsanity.com/paper/2306.09896/) (2023)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024)
- [TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark](https://mlsanity.com/paper/2410.00752/) (2024)
- [Long Code Arena: a Set of Benchmarks for Long-Context Code Models](https://mlsanity.com/paper/2406.11612/) (2024)
- [Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study](https://mlsanity.com/paper/2403.08604/) (2024)
- [TESTEVAL: Benchmarking Large Language Models for Test Case Generation](https://mlsanity.com/paper/2406.04531/) (2024)
- [On the Evaluation of Large Language Models in Unit Test Generation](https://mlsanity.com/paper/2406.18181/) (2024)
- [A Comparative Analysis of Large Language Models for Code Documentation Generation](https://mlsanity.com/paper/2312.10349/) (2023)
- [An Empirical Study on LLM-based Agents for Automated Bug Fixing](https://mlsanity.com/paper/2411.10213/) (2024)
- [TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration](https://mlsanity.com/paper/2408.03095/) (2024)
- [An Empirical Study on the Code Refactoring Capability of Large Language Models](https://mlsanity.com/paper/2411.02320/) (2024)
- [The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation](https://mlsanity.com/paper/2501.01329/) (2025)
- [TestBench: Evaluating Class-Level Test Case Generation Capability of Large Language Models](https://mlsanity.com/paper/2409.17561/) (2024)
- [LLM Test Generation via Iterative Hybrid Program Analysis](https://mlsanity.com/paper/2503.13580/) (2025)
- [CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification](https://mlsanity.com/paper/2502.08806/) (2025)
- [MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks](https://mlsanity.com/paper/2507.12284/) (2025)
- [CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?](https://mlsanity.com/paper/2412.02735/) (2024)

## Abstract (arXiv)

While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval.