# Automated Unit Test Improvement using Large Language Models at Meta

arXiv: https://arxiv.org/abs/2402.09171 · HTML: https://mlsanity.com/paper/2402.09171/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2024.

## Built on by (most cited first, 30 of 75)

- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents](https://mlsanity.com/paper/2504.08703/) (2025)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [CoverUp: Effective High Coverage Test Generation for Python](https://mlsanity.com/paper/2403.16218/) (2024)
- [Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation](https://mlsanity.com/paper/2502.14948/) (2025)
- [CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks](https://mlsanity.com/paper/2507.10535/) (2025)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [Mutation-Guided LLM-based Test Generation at Meta](https://mlsanity.com/paper/2501.12862/) (2025)
- [The Future of AI-Driven Software Engineering](https://mlsanity.com/paper/2406.07737/) (2024)
- [Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches](https://mlsanity.com/paper/2510.04905/) (2025)
- [debug-gym: A Text-Based Environment for Interactive Debugging](https://mlsanity.com/paper/2503.21557/) (2025)
- [CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation](https://mlsanity.com/paper/2502.10802/) (2025)
- [Migrating Code At Scale With LLMs At Google](https://mlsanity.com/paper/2504.09691/) (2025)
- [Metamorphic Testing of Large Language Models for Natural Language Processing](https://mlsanity.com/paper/2511.02108/) (2025)
- [Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests](https://mlsanity.com/paper/2408.11710/) (2024)
- [LLMs: A Game-Changer for Software Engineers?](https://mlsanity.com/paper/2411.00932/) (2024)
- [Learning to Generate Unit Test via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2508.21107/) (2025)
- [Rethinking the Evaluation of Secure Code Generation](https://mlsanity.com/paper/2503.15554/) (2025)
- [VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching](https://mlsanity.com/paper/2409.10756/) (2024)
- [LLM Test Generation via Iterative Hybrid Program Analysis](https://mlsanity.com/paper/2503.13580/) (2025)
- [CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification](https://mlsanity.com/paper/2502.08806/) (2025)
- [RAGProbe: An Automated Approach for Evaluating RAG Applications](https://mlsanity.com/paper/2409.19019/) (2024)
- [Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation](https://mlsanity.com/paper/2602.08146/) (2026)
- [Evaluating LLM-Based Regression Test Generation](https://mlsanity.com/paper/2501.11086/) (2025)
- [Are Coding Agents Generating Over-Mocked Tests? An Empirical Study](https://mlsanity.com/paper/2602.00409/) (2026)
- [Disproving Program Equivalence with LLMs](https://mlsanity.com/paper/2502.18473/) (2025)
- [How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench](https://mlsanity.com/paper/2507.02976/) (2025)
- [Automating REST API Postman Test Cases Using LLM](https://mlsanity.com/paper/2404.10678/) (2024)
- [Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Study](https://mlsanity.com/paper/2505.02502/) (2025)

## Abstract (arXiv)

This paper describes Meta's TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination. We describe the deployment of TestGen-LLM at Meta test-a-thons for the Instagram and Facebook platforms. In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. During Meta's Instagram and Facebook test-a-thons, it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers. We believe this is the first report on industrial scale deployment of LLM-generated code backed by such assurances of code improvement.