# TOGLL: Correct and Strong Test Oracle Generation with LLMs

arXiv: https://arxiv.org/abs/2405.03786 · HTML: https://mlsanity.com/paper/2405.03786/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2024.

## Built on by (most cited first, 24 of 24)

- [LeDex: Training LLMs to Better Self-Debug and Explain Code](https://mlsanity.com/paper/2405.18649/) (2024)
- [The Future of AI-Driven Software Engineering](https://mlsanity.com/paper/2406.07737/) (2024)
- [CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation](https://mlsanity.com/paper/2502.10802/) (2025)
- [Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents](https://mlsanity.com/paper/2505.05283/) (2025)
- [TestForge: Feedback-Driven, Agentic Test Suite Generation](https://mlsanity.com/paper/2503.14713/) (2025)
- [Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation](https://mlsanity.com/paper/2506.02943/) (2025)
- [Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning](https://mlsanity.com/paper/2603.15611/) (2026)
- [Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs](https://mlsanity.com/paper/2411.01789/) (2024)
- [Beyond Postconditions: Can Large Language Models infer Formal Contracts for Automatic Software Verification?](https://mlsanity.com/paper/2510.12702/) (2025)
- [Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis](https://mlsanity.com/paper/2510.26423/) (2025)
- [Navigating the growing field of research on AI for software testing -- the taxonomy for AI-augmented software testing and an ontology-driven literature survey](https://mlsanity.com/paper/2506.14640/) (2025)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [Understanding LLM-Driven Test Oracle Generation](https://mlsanity.com/paper/2601.05542/) (2026)
- [Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets](https://mlsanity.com/paper/2608.01000/) (2026)
- [Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges](https://mlsanity.com/paper/2504.16472/) (2025)
- [Just-in-Time Catching Test Generation at Meta](https://mlsanity.com/paper/2601.22832/) (2026)
- [Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus](https://mlsanity.com/paper/2603.29292/) (2026)
- [MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models](https://mlsanity.com/paper/2606.24815/) (2026)
- [Inferring Code Correctness from Specification](https://mlsanity.com/paper/2605.29822/) (2026)
- [Multi-Large Language Model Orchestrated Severity Assessment of Clinical Records (MOSAIC)](https://mlsanity.com/paper/2607.05032/) (2026)
- [Rethinking Code Performance Benchmarks for LLMs](https://mlsanity.com/paper/2607.07619/) (2026)
- [Fail-Aware and Explainable Test Oracle Prediction](https://mlsanity.com/paper/2607.11342/) (2026)
- [Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests](https://mlsanity.com/paper/2607.22883/) (2026)
- [Agent-Based Test Assertion Generation via Diverse Perspective Aggregation](https://mlsanity.com/paper/2608.05822/) (2026)

## Abstract (arXiv)

Test oracles play a crucial role in software testing, enabling effective bug detection. Despite initial promise, neural-based methods for automated test oracle generation often result in a large number of false positives and weaker test oracles. While LLMs have demonstrated impressive effectiveness in various software engineering tasks, including code generation, test case creation, and bug fixing, there remains a notable absence of large-scale studies exploring their effectiveness in test oracle generation. The question of whether LLMs can address the challenges in effective oracle generation is both compelling and requires thorough investigation. In this research, we present the first comprehensive study to investigate the capabilities of LLMs in generating correct, diverse, and strong test oracles capable of effectively identifying a large number of unique bugs. To this end, we fine-tuned seven code LLMs using six distinct prompts on the SF110 dataset. Utilizing the most effective fine-tuned LLM and prompt pair, we introduce TOGLL, a novel LLM-based method for test oracle generation. To investigate the generalizability of TOGLL, we conduct studies on 25 large-scale Java projects. Besides assessing the correctness, we also assess the diversity and strength of the generated oracles. We compare the results against EvoSuite and the state-of-the-art neural method, TOGA. Our findings reveal that TOGLL can produce 3.8 times more correct assertion oracles and 4.9 times more exception oracles. Moreover, our findings demonstrate that TOGLL is capable of generating significantly diverse test oracles. It can detect 1,023 unique bugs that EvoSuite cannot, which is ten times more than what the previous SOTA neural-based method, TOGA, can detect.