# Just-in-Time Catching Test Generation at Meta

arXiv: https://arxiv.org/abs/2601.22832 · HTML: https://mlsanity.com/paper/2601.22832/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 75 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [The Art, Science, and Engineering of Fuzzing: A Survey](https://mlsanity.com/paper/1812.00140/) (2018)
- [Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study](https://mlsanity.com/paper/2310.02059/) (2023)
- [Mutation-Guided LLM-based Test Generation at Meta](https://mlsanity.com/paper/2501.12862/) (2025)
- [TOGLL: Correct and Strong Test Oracle Generation with LLMs](https://mlsanity.com/paper/2405.03786/) (2024)
- [Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions](https://mlsanity.com/paper/2402.08967/) (2024)
- [WES: Agent-based User Interaction Simulation on Real Infrastructure](https://mlsanity.com/paper/2004.05363/) (2020)
- [Perfect is the enemy of test oracle](https://mlsanity.com/paper/2302.01488/) (2023)
- [Moving Faster and Reducing Risk: Using LLMs in Release Deployment](https://mlsanity.com/paper/2410.06351/) (2024)
- [Observation-based unit test generation at Meta](https://mlsanity.com/paper/2402.06111/) (2024)
- [Enhancing Testing at Meta with Rich-State Simulated Populations](https://mlsanity.com/paper/2403.15374/) (2024)
- [Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges](https://mlsanity.com/paper/2504.16472/) (2025)

## Built on by (most cited first, 1 of 1)

- [Multi-Large Language Model Orchestrated Severity Assessment of Clinical Records (MOSAIC)](https://mlsanity.com/paper/2607.05032/) (2026)

## Abstract (arXiv)

We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hardening tests, which pass at generation time, catching tests are meant to fail, surfacing bugs before code lands. The primary challenge is to reduce development drag from false positive test failures. Analyzing 22,126 generated tests, we show code-change-aware methods improve candidate catch generation 4x over hardening tests and 20x over coincidentally failing tests. To address false positives, we use rule-based and LLM-based assessors. These assessors reduce human review load by 70%. Inferential statistical analysis showed that human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives. We reported 41 candidate catches to engineers; 8 were confirmed to be true positives, 4 of which would have led to serious failures had they remained uncaught. Overall, our results show that Just-in-Time catching is scalable, industrially applicable, and that it prevents serious failures from reaching production.