# TOGA: A Neural Method for Test Oracle Generation

arXiv: https://arxiv.org/abs/2109.09262 · HTML: https://mlsanity.com/paper/2109.09262/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2021.

## Built on by (most cited first, 30 of 57)

- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?](https://mlsanity.com/paper/2310.01831/) (2023)
- [Exploring and Unleashing the Power of Large Language Models in Automated Code Translation](https://mlsanity.com/paper/2404.14646/) (2024)
- [Interactive Code Generation via Test-Driven User-Intent Formalization](https://mlsanity.com/paper/2208.05950/) (2022)
- [ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers](https://mlsanity.com/paper/2305.14591/) (2023)
- [Learning Deep Semantics for Test Completion](https://mlsanity.com/paper/2302.10166/) (2023)
- [Repairing Bugs in Python Assignments Using Large Language Models](https://mlsanity.com/paper/2209.14876/) (2022)
- [CAT-LM: Training Language Models on Aligned Code And Tests](https://mlsanity.com/paper/2310.01602/) (2023)
- [SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?](https://mlsanity.com/paper/2606.07682/) (2026)
- [Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation](https://mlsanity.com/paper/2310.02368/) (2023)
- [Evaluating LLM-driven User-Intent Formalization for Verification-Aware Languages](https://mlsanity.com/paper/2406.09757/) (2024)
- [CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation](https://mlsanity.com/paper/2502.10802/) (2025)
- [Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools](https://mlsanity.com/paper/2312.10622/) (2023)
- [The Program Testing Ability of Large Language Models for Code](https://mlsanity.com/paper/2310.05727/) (2023)
- [TestForge: Feedback-Driven, Agentic Test Suite Generation](https://mlsanity.com/paper/2503.14713/) (2025)
- [Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation](https://mlsanity.com/paper/2506.02943/) (2025)
- [Intent Formalization: A Grand Challenge for Reliable Coding in the Age of AI Agents](https://mlsanity.com/paper/2603.17150/) (2026)
- [Domain Adaptation for Code Model-based Unit Test Case Generation](https://mlsanity.com/paper/2308.08033/) (2023)
- [Assessing Evaluation Metrics for Neural Test Oracle Generation](https://mlsanity.com/paper/2310.07856/) (2023)
- [On the Effectiveness of Large Language Models in Writing Alloy Formulas](https://mlsanity.com/paper/2502.15441/) (2025)
- [Automating the Correctness Assessment of AI-generated Code for Security Contexts](https://mlsanity.com/paper/2310.18834/) (2023)
- [Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs](https://mlsanity.com/paper/2411.01789/) (2024)
- [RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practice](https://mlsanity.com/paper/2502.14930/) (2025)
- [Beyond Postconditions: Can Large Language Models infer Formal Contracts for Automatic Software Verification?](https://mlsanity.com/paper/2510.12702/) (2025)
- [Revisiting and Improving Retrieval-Augmented Deep Assertion Generation](https://mlsanity.com/paper/2309.10264/) (2023)
- [UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testing](https://mlsanity.com/paper/2402.03396/) (2024)
- [exLong: Generating Exceptional Behavior Tests with Large Language Models](https://mlsanity.com/paper/2405.14619/) (2024)
- [Learning Program Behavioral Models from Synthesized Input-Output Pairs](https://mlsanity.com/paper/2407.08597/) (2024)
- [ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code](https://mlsanity.com/paper/2411.16927/) (2024)

## Abstract (arXiv)

Testing is widely recognized as an important stage of the software development lifecycle. Effective software testing can provide benefits such as bug finding, preventing regressions, and documentation. In terms of documentation, unit tests express a unit's intended functionality, as conceived by the developer. A test oracle, typically expressed as an condition, documents the intended behavior of a unit under a given test prefix. Synthesizing a functional test oracle is a challenging problem, as it must capture the intended functionality rather than the implemented functionality. In this paper, we propose TOGA (a neural method for Test Oracle GenerAtion), a unified transformer-based neural approach to infer both exceptional and assertion test oracles based on the context of the focal method. Our approach can handle units with ambiguous or missing documentation, and even units with a missing implementation. We evaluate our approach on both oracle inference accuracy and functional bug-finding. Our technique improves accuracy by 33% over existing oracle inference approaches, achieving 96% overall accuracy on a held out test dataset. Furthermore, we show that when integrated with a automated test generation tool (EvoSuite), our approach finds 57 real world bugs in large-scale Java programs, including 30 bugs that are not found by any other automated testing method in our evaluation.