# Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing

arXiv: https://arxiv.org/abs/1808.05353 · HTML: https://mlsanity.com/paper/1808.05353/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2018.

## Built on by (most cited first, 30 of 31)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [MTTM: Metamorphic Testing for Textual Content Moderation Software](https://mlsanity.com/paper/2302.05706/) (2023)
- [New Job, New Gender? Measuring the Social Bias in Image Generation Models](https://mlsanity.com/paper/2401.00763/) (2024)
- [MDPFuzz: Testing Models Solving Markov Decision Processes](https://mlsanity.com/paper/2112.02807/) (2021)
- [If a Human Can See It, So Should Your System: Reliability Requirements for Machine Vision Components](https://mlsanity.com/paper/2202.03930/) (2022)
- [A Survey on Verification and Validation, Testing and Evaluations of Neurosymbolic Artificial Intelligence](https://mlsanity.com/paper/2401.03188/) (2024)
- [Automatic Fault Detection for Deep Learning Programs Using Graph Transformations](https://mlsanity.com/paper/2105.08095/) (2021)
- ["Oops, Did I Just Say That?" Testing and Repairing Unethical Suggestions of Large Language Models with Suggest-Critique-Reflect Process](https://mlsanity.com/paper/2305.02626/) (2023)
- [Towards a Change Taxonomy for Machine Learning Systems](https://mlsanity.com/paper/2203.11365/) (2022)
- [Perfect is the enemy of test oracle](https://mlsanity.com/paper/2302.01488/) (2023)
- [Software Testing for Machine Learning](https://mlsanity.com/paper/2205.00210/) (2022)
- [Specifying and Testing k-Safety Properties for Machine-Learning Models](https://mlsanity.com/paper/2206.06054/) (2022)
- [Validating Multimedia Content Moderation Software via Semantic Fusion](https://mlsanity.com/paper/2305.13623/) (2023)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [Rule-Guided Reinforcement Learning Policy Evaluation and Improvement](https://mlsanity.com/paper/2503.09270/) (2025)
- [Semantic-Based Neural Network Repair](https://mlsanity.com/paper/2306.07995/) (2023)
- [An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software](https://mlsanity.com/paper/2308.09810/) (2023)
- [Metamorphic Testing for Pose Estimation Systems](https://mlsanity.com/paper/2502.09460/) (2025)
- [Semantic Invariance in Agentic AI](https://mlsanity.com/paper/2603.13173/) (2026)
- [ML4ML: Automated Invariance Testing for Machine Learning Models](https://mlsanity.com/paper/2109.12926/) (2021)
- [Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques](https://mlsanity.com/paper/2202.12139/) (2022)
- [Testing Feedforward Neural Networks Training Programs](https://mlsanity.com/paper/2204.00694/) (2022)
- [MetaRAG: Metamorphic Testing for Hallucination Detection in RAG Systems](https://mlsanity.com/paper/2509.09360/) (2025)
- [Provably Valid and Diverse Mutations of Real-World Media Data for DNN Testing](https://mlsanity.com/paper/2112.01956/) (2021)
- [ExAIS: Executable AI Semantics](https://mlsanity.com/paper/2202.09868/) (2022)
- [TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systems](https://mlsanity.com/paper/2507.05932/) (2025)
- [MT4DP: Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing](https://mlsanity.com/paper/2507.11092/) (2025)
- [Metamorphic Testing for Audio Content Moderation Software](https://mlsanity.com/paper/2509.24215/) (2025)
- [Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning](https://mlsanity.com/paper/2606.06056/) (2026)
- [Metamorphic Testing for Clinical ML Models: A Framework Proposal and Pilot Study](https://mlsanity.com/paper/2607.22984/) (2026)

## Abstract (arXiv)

We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, testing such applications is extremely challenging and would be very expensive if we follow today's methodologies. In this work, we present an articulation of the challenges in testing ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.