# Mutation-Guided LLM-based Test Generation at Meta

arXiv: https://arxiv.org/abs/2501.12862 · HTML: https://mlsanity.com/paper/2501.12862/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 66 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 16 of 16)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [LLMorpheus: Mutation Testing using Large Language Models](https://mlsanity.com/paper/2404.09952/) (2024)
- [Perfect is the enemy of test oracle](https://mlsanity.com/paper/2302.01488/) (2023)
- [Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs](https://mlsanity.com/paper/2401.05940/) (2024)
- [Large Language Models for Equivalent Mutant Detection: How Far Are We?](https://mlsanity.com/paper/2408.01760/) (2024)
- [Observation-based unit test generation at Meta](https://mlsanity.com/paper/2402.06111/) (2024)
- [Finding Missed Code Size Optimizations in Compilers using LLMs](https://mlsanity.com/paper/2501.00655/) (2024)
- [Practical Mutation Testing at Scale](https://mlsanity.com/paper/2102.11378/) (2021)
- [Keeping Mutation Test Suites Consistent and Relevant with Long-Standing Mutants](https://mlsanity.com/paper/2212.11762/) (2022)

## Built on by (most cited first, 25 of 25)

- [Hallucination Detection in Large Language Models with Metamorphic Relations](https://mlsanity.com/paper/2502.15844/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Learning to Generate Unit Test via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2508.21107/) (2025)
- [Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation](https://mlsanity.com/paper/2602.08146/) (2026)
- [Are Coding Agents Generating Over-Mocked Tests? An Empirical Study](https://mlsanity.com/paper/2602.00409/) (2026)
- [AI-Driven Tools in Modern Software Quality Assurance: An Assessment of Benefits, Challenges, and Future Directions](https://mlsanity.com/paper/2506.16586/) (2025)
- [EvoGPT: Leveraging LLM-Driven Seed Diversity to Improve Search-Based Test Suite Generation](https://mlsanity.com/paper/2505.12424/) (2025)
- [textttReMind: Understanding Deductive Code Reasoning in LLMs](https://mlsanity.com/paper/2511.00488/) (2025)
- [RM -RF: Reward Model for Run-Free Unit Test Evaluation](https://mlsanity.com/paper/2601.13097/) (2026)
- [Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges](https://mlsanity.com/paper/2504.16472/) (2025)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Just-in-Time Catching Test Generation at Meta](https://mlsanity.com/paper/2601.22832/) (2026)
- [PBT-Bench: Benchmarking AI Agents on Property-Based Testing](https://mlsanity.com/paper/2605.15229/) (2026)
- [Towards Knowledge Alignment in Code LLMs: Contrastive Unlearning for Evolving APIs](https://mlsanity.com/paper/2606.30810/) (2026)
- [Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?](https://mlsanity.com/paper/2607.01211/) (2026)
- [Inside the Skill Market: From Software Engineering Activities to Reusable Agent Skills](https://mlsanity.com/paper/2607.09065/) (2026)
- [Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents](https://mlsanity.com/paper/2610.01023/) (2026)
- [Multi-Agent Coordinated Rename Refactoring](https://mlsanity.com/paper/2601.00482/) (2026)
- [LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB](https://mlsanity.com/paper/2604.14437/) (2026)
- [Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics](https://mlsanity.com/paper/2604.17715/) (2026)
- [MuMuTestUp: Mutation-based Multi-Agent Test Case Update](https://mlsanity.com/paper/2605.19265/) (2026)
- [Automated Vulnerability Injection in Smart Contracts Using Large Language Models](https://mlsanity.com/paper/2609.02624/) (2026)
- [VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories](https://mlsanity.com/paper/2609.05978/) (2026)
- [Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles](https://mlsanity.com/paper/2609.22220/) (2026)
- [Scalable Weak-to-Strong Verifier Hardening for Agent Environments](https://mlsanity.com/paper/2610.15924/) (2026)

## Abstract (arXiv)

This paper describes Meta's ACH system for mutation-guided LLM-based test generation. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation testing. Instead, it focuses on generating currently undetected faults that are specific to an issue of concern. From these currently uncaught faults, ACH generates tests that can catch them, thereby `killing' the mutants and consequently hardening the platform against regressions. We use privacy concerns to illustrate our approach, but ACH can harden code against em any type of regression. In total, ACH was applied to 10,795 Android Kotlin classes in 7 software platforms deployed by Meta, from which it generated 9,095 mutants and 571 privacy-hardening test cases. ACH also deploys an LLM-based equivalent mutant detection agent that achieves a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing). ACH was used by Messenger and WhatsApp test-a-thons where engineers accepted 73% of its tests, judging 36% to privacy relevant. We conclude that ACH hardens code against specific concerns and that, even when its tests do not directly tackle the specific concern, engineers find them useful for their other benefits.