# A Review on Oracle Issues in Machine Learning

arXiv: https://arxiv.org/abs/2105.01407 · HTML: https://mlsanity.com/paper/2105.01407/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 53 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 9 of 9)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1709.06683/) (2017)
- [3D Object Reconstruction from a Single Depth View with Adversarial Learning](https://mlsanity.com/paper/1708.07969/) (2017)
- [Probabilistic Machine Learning for Healthcare](https://mlsanity.com/paper/2009.11087/) (2020)
- [Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defects](https://mlsanity.com/paper/2009.01521/) (2020)

## Built on by (most cited first, 1 of 1)

- [Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testing](https://mlsanity.com/paper/2207.10495/) (2022)

## Abstract (arXiv)

Machine learning contrasts with traditional software development in that the oracle is the data, and the data is not always a correct representation of the problem that machine learning tries to model. We present a survey of the oracle issues found in machine learning and state-of-the-art solutions for dealing with these issues. These include lines of research for differential testing, metamorphic testing, and test coverage. We also review some recent improvements to robustness during modeling that reduce the impact of oracle issues, as well as tools and frameworks for assisting in testing and discovering issues specific to the dataset.