# Test Oracle Automation in the era of LLMs

arXiv: https://arxiv.org/abs/2405.12766 · HTML: https://mlsanity.com/paper/2405.12766/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2024.

## Built on by (most cited first, 11 of 11)

- [PromptPex: Automatic Test Generation for Language Model Prompts](https://mlsanity.com/paper/2503.05070/) (2025)
- [Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy](https://mlsanity.com/paper/2503.00481/) (2025)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [Understanding LLM-Driven Test Oracle Generation](https://mlsanity.com/paper/2601.05542/) (2026)
- [LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle](https://mlsanity.com/paper/2607.06195/) (2026)
- [Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges](https://mlsanity.com/paper/2504.16472/) (2025)
- [Test-driven Software Experimentation with LASSO: an LLM Prompt Benchmarking Example](https://mlsanity.com/paper/2410.08911/) (2024)
- [Multi-Large Language Model Orchestrated Severity Assessment of Clinical Records (MOSAIC)](https://mlsanity.com/paper/2607.05032/) (2026)
- [Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs](https://mlsanity.com/paper/2607.12273/) (2026)
- [SenWorld: A Digital-Twin Simulation for Generating Context-Rich Evaluation Data](https://mlsanity.com/paper/2607.19949/) (2026)
- [ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks](https://mlsanity.com/paper/2609.18805/) (2026)

## Abstract (arXiv)

The effectiveness of a test suite in detecting faults highly depends on the correctness and completeness of its test oracles. Large Language Models (LLMs) have already demonstrated remarkable proficiency in tackling diverse software testing tasks, such as automated test generation and program repair. This paper aims to enable discussions on the potential of using LLMs for test oracle automation, along with the challenges that may emerge during the generation of various types of oracles. Additionally, our aim is to initiate discussions on the primary threats that SE researchers must consider when employing LLMs for oracle automation, encompassing concerns regarding oracle deficiencies and data leakages.