# METTLE: a METamorphic testing approach to assessing and validating unsupervised machine LEarning systems

arXiv: https://arxiv.org/abs/1807.10453 · HTML: https://mlsanity.com/paper/1807.10453/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2018.

## Built on by (most cited first, 8 of 8)

- [AequeVox: Automated Fairness Testing of Speech Recognition Systems](https://mlsanity.com/paper/2110.09843/) (2021)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [Robustness Evaluation in Hand Pose Estimation Models using Metamorphic Testing](https://mlsanity.com/paper/2303.04566/) (2023)
- [Use of Metamorphic Relations as Knowledge Carriers to Train Deep Neural Networks](https://mlsanity.com/paper/2104.04718/) (2021)
- [A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?](https://mlsanity.com/paper/2511.05476/) (2025)
- [TimeGym: Debugging for Time Series Modeling in Python](https://mlsanity.com/paper/2105.01404/) (2021)
- [Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers](https://mlsanity.com/paper/2410.22663/) (2024)
- [Evaluating and Enhancing Segmentation Model Robustness with Metamorphic Testing](https://mlsanity.com/paper/2504.02335/) (2025)

## Abstract (arXiv)

Unsupervised machine learning is the training of an artificial intelligence system using information that is neither classified nor labeled, with a view to modeling the underlying structure or distribution in a dataset. Since unsupervised machine learning systems are widely used in many real-world applications, assessing the appropriateness of these systems and validating their implementations with respect to individual users' requirements and specific application scenarios,/,contexts are indisputably two important tasks. Such assessment and validation tasks, however, are fairly challenging due to the absence of a priori knowledge of the data. In view of this challenge, we develop a METamorphic Testing approach to assessing and validating unsupervised machine LEarning systems, abbreviated as METTLE. Our approach provides a new way to unveil the (possibly latent) characteristics of various machine learning systems, by explicitly considering the specific expectations and requirements of these systems from individual users' perspectives. To support METTLE, we have further formulated 11 generic metamorphic relations (MRs), covering users' generally expected characteristics that should be possessed by machine learning systems. To demonstrate the viability and effectiveness of METTLE we have performed an experiment involving six commonly used clustering systems. Our experiment has shown that, guided by user-defined MR-based adequacy criteria, end users are able to assess, validate, and select appropriate clustering systems in accordance with their own specific needs. Our investigation has also yielded insightful understanding and interpretation of the behavior of the machine learning systems from an end-user software engineering's perspective, rather than a designer's or implementor's perspective, who normally adopts a theoretical approach.