# Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges

arXiv: https://arxiv.org/abs/2504.16472 · HTML: https://mlsanity.com/paper/2504.16472/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 90 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 18 of 18)

- [Online Learning: A Comprehensive Survey](https://mlsanity.com/paper/1802.02871/) (2018)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024)
- [The Art, Science, and Engineering of Fuzzing: A Survey](https://mlsanity.com/paper/1812.00140/) (2018)
- [TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark](https://mlsanity.com/paper/2410.00752/) (2024)
- [HITS: High-coverage LLM-based Unit Test Generation via Method Slicing](https://mlsanity.com/paper/2408.11324/) (2024)
- [Mutation-Guided LLM-based Test Generation at Meta](https://mlsanity.com/paper/2501.12862/) (2025)
- [TOGLL: Correct and Strong Test Oracle Generation with LLMs](https://mlsanity.com/paper/2405.03786/) (2024)
- [Automatic Code Summarization: A Systematic Literature Review](https://mlsanity.com/paper/1909.04352/) (2019)
- [Test Oracle Automation in the era of LLMs](https://mlsanity.com/paper/2405.12766/) (2024)
- [Do LLMs generate test oracles that capture the actual or the expected program behaviour?](https://mlsanity.com/paper/2410.21136/) (2024)
- [WES: Agent-based User Interaction Simulation on Real Infrastructure](https://mlsanity.com/paper/2004.05363/) (2020)
- [Observation-based unit test generation at Meta](https://mlsanity.com/paper/2402.06111/) (2024)
- [Developer-Centric Test Amplification The Interplay Between Automatic Generation and Human Exploration](https://mlsanity.com/paper/2108.12249/) (2021)
- [Enhancing Testing at Meta with Rich-State Simulated Populations](https://mlsanity.com/paper/2403.15374/) (2024)
- [Mimicking Production Behavior with Generated Mocks](https://mlsanity.com/paper/2208.01321/) (2022)

## Built on by (most cited first, 1 of 1)

- [Just-in-Time Catching Test Generation at Meta](https://mlsanity.com/paper/2601.22832/) (2026)

## Abstract (arXiv)

Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software test generation. More specifically, we formally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect against future regressions, while a catching test is one that catches such a regression or a fault in new functionality introduced by a code change. Hardening tests can be generated at any time and may become catching tests when a future regression is caught. We also define and motivate the Catching 'Just-in-Time' (JiTTest) Challenge, in which tests are generated 'just-in-time' to catch new faults before they land into production. We show that any solution to Catching JiTTest generation can also be repurposed to catch latent faults in legacy code. We enumerate possible outcomes for hardening and catching tests and JiTTests, and discuss open research problems, deployment options, and initial results from our work on automated LLM-based hardening at Meta. This paper was written to accompany the keynote by the authors at the ACM International Conference on the Foundations of Software Engineering (FSE) 2025. Author order is alphabetical. The corresponding author is Mark Harman.