mlsanityData to 7 Oct 2026

Metamorphic Testing: A New Approach for Generating Next Test Cases

arXiv 2002.12543 ↗First posted 28 Feb 2020cs.SEcited by 71

Not on the map. Why?

Abstract (arXiv)

In software testing, a set of test cases is constructed according to some predefined selection criteria. The software is then examined against these test cases. Three interesting observations have been made on the current artifacts of software testing. Firstly, an error-revealing test case is considered useful while a successful test case which does not reveal software errors is usually not further investigated. Whether these successful test cases still contain useful information for revealing software errors has not been properly studied. Secondly, no matter how extensive the testing has been conducted in the development phase, errors may still exist in the software [5]. These errors, if left undetected, may eventually cause damage to the production system. The study of techniques for uncovering software errors in the production phase is seldom addressed in the literature. Thirdly, as indicated by Weyuker in [6], the availability of test oracles is pragmatically unattainable in most situations. However, the availability of test oracles is generally assumed in conventional software testing techniques. In this paper, we propose a novel test case selection technique that derives new test cases from the successful ones. The selection aims at revealing software errors that are possibly left undetected in successful test cases which may be generated using some existing strategies. As such, the proposed technique augments the effectiveness of existing test selection strategies. The technique also helps uncover software errors in the production phase and can be used in the absence of test oracles.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 4; 2023: 10; 2024: 11; 2025: 14; 2026: 27.

71 papers.

Sort:

By area:

  1. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  2. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  3. 2022Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Codecited by 39
  4. 2021How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Reviewcited by 36
  5. 2026ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditionscited by 31
  6. 2024Consistency Checks for Language Model Forecasterscited by 27
  7. 2024The Future of AI-Driven Software Engineeringcited by 23
  8. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 20
  9. 2024New Job, New Gender? Measuring the Social Bias in Image Generation Modelscited by 15
  10. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  11. 2026ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimizationcited by 8
  12. 2024MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systemscited by 7
38 more
  1. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Modelscited by 6
  2. 2021Using Metamorphic Relations to Verify and Enhance Artcode Classificationcited by 5
  3. 2022Specifying and Testing k-Safety Properties for Machine-Learning Modelscited by 5
  4. 2023Validating Multimedia Content Moderation Software via Semantic Fusioncited by 5
  5. 2024MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generationcited by 4
  6. 2024Deep Learning Library Testing: Definition, Methods and Challengescited by 4
  7. 2026Metamorphic Testing of Vision-Language Action-Enabled Robotscited by 4
  8. 2023An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Softwarecited by 3
  9. 2023Automatically Testing Functional Properties of Code Translation Modelscited by 3
  10. 2023On the Potential and Limitations of Few-Shot In-Context Learning to Generate Metamorphic Specifications for Tax Preparation Softwarecited by 3
  11. 2024An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarioscited by 3
  12. 2024Software Engineering Methods For AI-Driven Deductive Legal Reasoningcited by 3
  13. 2024Evaluating Human Trajectory Prediction with Metamorphic Testingcited by 3
  14. 2026Memory Reward Inflation in Self-Improving LLM Agentscited by 3
  15. 2022Sibylvariant Transformations for Robust Text Classificationcited by 2
  16. 2023Retromorphic Testing: A New Approach to the Test Oracle Problemcited by 2
  17. 2024Machine Translation Testing via Syntactic Tree Pruningcited by 2
  18. 2024Computer Vision Intelligence Test Modeling and Generation: A Case Study on Smart OCRcited by 2
  19. 2025Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learningcited by 2
  20. 2025MetaRAG: Metamorphic Testing for Hallucination Detection in RAG Systemscited by 2
  21. 2025AutoMT: A Multi-Agent LLM Framework for Automated Metamorphic Testing of Autonomous Driving Systemscited by 2
  22. 2026Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformationscited by 2
  23. 2026Semantic Invariance in Agentic AIcited by 2
  24. 2021Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testingcited by 1
  25. 2021Finding Deviated Behaviors of the Compressed DNN Models for Image Classificationscited by 1
  26. 2022Is Neuron Coverage Needed to Make Person Detection More Robust?cited by 1
  27. 2025MetaLogic: Robustness Evaluation of Text-to-Image Models via Logically Equivalent Promptscited by 1
  28. 2025A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?cited by 1
  29. 2025LLM Assisted Coding with Metamorphic Specification Mutation Agentcited by 1
  30. 2025Beyond Accuracy: A Geometric Stability Analysis of Large Language Models in Chess Evaluationcited by 1
  31. 2026An Axiomatic Benchmark for Evaluation of Scientific Novelty Metricscited by 1
  32. 2026InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?cited by 1
  33. 2026Social Bias in LLM-Generated Code: Benchmark and Mitigationcited by 1
  34. 2026AgentModernize: Preserving Business Logic in Legacy Modernization with Multi-Agent LLMs and Behavioral Specification Graphscited by 1
  35. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systemscited by 1
  36. 2026Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agentscited by 1
  37. 2021HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networkscited by 0
  38. 2023ASTER: Automatic Speech Recognition System Accessibility Testing for Stuttererscited by 0

50 most cited of 71 All 71 (JSON)