# Metamorphic Testing: A New Approach for Generating Next Test Cases

arXiv: https://arxiv.org/abs/2002.12543 · HTML: https://mlsanity.com/paper/2002.12543/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2020.

## Built on by (most cited first, 30 of 75)

- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [Evaluating Superhuman Models with Consistency Checks](https://mlsanity.com/paper/2306.09983/) (2023)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code](https://mlsanity.com/paper/2206.01335/) (2022)
- [ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions](https://mlsanity.com/paper/2601.06112/) (2026)
- [Consistency Checks for Language Model Forecasters](https://mlsanity.com/paper/2412.18544/) (2024)
- [The Future of AI-Driven Software Engineering](https://mlsanity.com/paper/2406.07737/) (2024)
- [MTTM: Metamorphic Testing for Textual Content Moderation Software](https://mlsanity.com/paper/2302.05706/) (2023)
- [New Job, New Gender? Measuring the Social Bias in Image Generation Models](https://mlsanity.com/paper/2401.00763/) (2024)
- [Metamorphic Testing of Large Language Models for Natural Language Processing](https://mlsanity.com/paper/2511.02108/) (2025)
- [ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization](https://mlsanity.com/paper/2602.15983/) (2026)
- [MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems](https://mlsanity.com/paper/2412.15557/) (2024)
- [VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models](https://mlsanity.com/paper/2312.04087/) (2023)
- [Using Metamorphic Relations to Verify and Enhance Artcode Classification](https://mlsanity.com/paper/2108.02694/) (2021)
- [Specifying and Testing k-Safety Properties for Machine-Learning Models](https://mlsanity.com/paper/2206.06054/) (2022)
- [Validating Multimedia Content Moderation Software via Semantic Fusion](https://mlsanity.com/paper/2305.13623/) (2023)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [Metamorphic Testing of Vision-Language Action-Enabled Robots](https://mlsanity.com/paper/2602.22579/) (2026)
- [MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation](https://mlsanity.com/paper/2402.14480/) (2024)
- [An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software](https://mlsanity.com/paper/2308.09810/) (2023)
- [Automatically Testing Functional Properties of Code Translation Models](https://mlsanity.com/paper/2309.12813/) (2023)
- [On the Potential and Limitations of Few-Shot In-Context Learning to Generate Metamorphic Specifications for Tax Preparation Software](https://mlsanity.com/paper/2311.11979/) (2023)
- [An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios](https://mlsanity.com/paper/2402.01723/) (2024)
- [Software Engineering Methods For AI-Driven Deductive Legal Reasoning](https://mlsanity.com/paper/2404.09868/) (2024)
- [Evaluating Human Trajectory Prediction with Metamorphic Testing](https://mlsanity.com/paper/2407.18756/) (2024)
- [Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations](https://mlsanity.com/paper/2602.05523/) (2026)
- [Semantic Invariance in Agentic AI](https://mlsanity.com/paper/2603.13173/) (2026)
- [Memory Reward Inflation in Self-Improving LLM Agents](https://mlsanity.com/paper/2608.00017/) (2026)
- [Sibylvariant Transformations for Robust Text Classification](https://mlsanity.com/paper/2205.05137/) (2022)
- [Retromorphic Testing: A New Approach to the Test Oracle Problem](https://mlsanity.com/paper/2310.06433/) (2023)

## Abstract (arXiv)

In software testing, a set of test cases is constructed according to some predefined selection criteria. The software is then examined against these test cases. Three interesting observations have been made on the current artifacts of software testing. Firstly, an error-revealing test case is considered useful while a successful test case which does not reveal software errors is usually not further investigated. Whether these successful test cases still contain useful information for revealing software errors has not been properly studied. Secondly, no matter how extensive the testing has been conducted in the development phase, errors may still exist in the software [5]. These errors, if left undetected, may eventually cause damage to the production system. The study of techniques for uncovering software errors in the production phase is seldom addressed in the literature. Thirdly, as indicated by Weyuker in [6], the availability of test oracles is pragmatically unattainable in most situations. However, the availability of test oracles is generally assumed in conventional software testing techniques. In this paper, we propose a novel test case selection technique that derives new test cases from the successful ones. The selection aims at revealing software errors that are possibly left undetected in successful test cases which may be generated using some existing strategies. As such, the proposed technique augments the effectiveness of existing test selection strategies. The technique also helps uncover software errors in the production phase and can be used in the absence of test oracles.