# Do LLMs generate test oracles that capture the actual or the expected program behaviour?

arXiv: https://arxiv.org/abs/2410.21136 · HTML: https://mlsanity.com/paper/2410.21136/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2024.

## Built on by (most cited first, 11 of 11)

- [LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops](https://mlsanity.com/paper/2603.23613/) (2026)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [Understanding LLM-Driven Test Oracle Generation](https://mlsanity.com/paper/2601.05542/) (2026)
- [LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle](https://mlsanity.com/paper/2607.06195/) (2026)
- [Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets](https://mlsanity.com/paper/2608.01000/) (2026)
- [Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges](https://mlsanity.com/paper/2504.16472/) (2025)
- [All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code](https://mlsanity.com/paper/2606.18168/) (2026)
- [RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements](https://mlsanity.com/paper/2604.25862/) (2026)
- [Inferring Code Correctness from Specification](https://mlsanity.com/paper/2605.29822/) (2026)
- [The Quality of Claude AI-authored Python Tests Is Not Weaker Than Human-authored Tests](https://mlsanity.com/paper/2608.15188/) (2026)
- [Independent Verification Paths Are Not Independent: A Case Study of Common-Mode Failure in a Satellite Catalogue Pipeline](https://mlsanity.com/paper/2609.37603/) (2026)

## Abstract (arXiv)

Software testing is an essential part of the software development cycle to improve the code quality. Typically, a unit test consists of a test prefix and a test oracle which captures the developer's intended behaviour. A known limitation of traditional test generation techniques (e.g. Randoop and Evosuite) is that they produce test oracles that capture the actual program behaviour rather than the expected one. Recent approaches leverage Large Language Models (LLMs), trained on an enormous amount of data, to generate developer-like code and test cases. We investigate whether the LLM-generated test oracles capture the actual or expected software behaviour. We thus, conduct a controlled experiment to answer this question, by studying LLMs performance on two tasks, namely, test oracle classification and generation. The study includes developer-written and automatically generated test cases and oracles for 24 open-source Java repositories, and different well tested prompts. Our findings show that LLM-based test generation approaches are also prone on generating oracles that capture the actual program behaviour rather than the expected one. Moreover, LLMs are better at generating test oracles rather than classifying the correct ones, and can generate better test oracles when the code contains meaningful test or variable names. Finally, LLM-generated test oracles have higher fault detection potential than the Evosuite ones.