# Using Machine Learning to Generate Test Oracles: A Systematic Literature Review

arXiv: https://arxiv.org/abs/2107.00906 · HTML: https://mlsanity.com/paper/2107.00906/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2021.

## Built on by (most cited first, 7 of 7)

- [Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs](https://mlsanity.com/paper/2411.01789/) (2024)
- [Fault Localization for Buggy Deep Learning Framework Conversions in Image Recognition](https://mlsanity.com/paper/2306.06157/) (2023)
- [The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study](https://mlsanity.com/paper/2206.10210/) (2022)
- [Automated Unit Test Case Generation: A Systematic Literature Review](https://mlsanity.com/paper/2504.20357/) (2025)
- [Learning Correct Behavior from Examples: Validating Sequential Execution in Autonomous Agents](https://mlsanity.com/paper/2605.03159/) (2026)
- [Mapping the Structure and Evolution of Software Testing Research Over the Past Three Decades](https://mlsanity.com/paper/2109.04086/) (2021)
- [Automated Support for Unit Test Generation: A Tutorial Book Chapter](https://mlsanity.com/paper/2110.13575/) (2021)

## Abstract (arXiv)

Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field. Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and - most commonly - expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions or code metadata - including neural networks, support vector machines, adaptive boosting, and decision trees. Oracles are evaluated using the mutation score, correct classifications, accuracy, and ROC. Work-to-date show great promise, but there are significant open challenges regarding the requirements imposed on training data, the complexity of modeled functions, the ML algorithms employed - and how they are applied - the benchmarks used by researchers, and replicability of the studies. We hope that our findings will serve as a roadmap and inspiration for researchers in this field.