# Metric Learning Improves the Ability of Combinatorial Coverage Metrics to Anticipate Classification Error

arXiv: https://arxiv.org/abs/2302.14616 · HTML: https://mlsanity.com/paper/2302.14616/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 30 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 8 of 8)

- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://mlsanity.com/paper/2002.08473/) (2020)
- [Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components](https://mlsanity.com/paper/1804.06760/) (2018)
- [metric-learn: Metric Learning Algorithms in Python](https://mlsanity.com/paper/1908.04710/) (2019)
- [FreaAI: Automated extraction of data slices to test machine learning models](https://mlsanity.com/paper/2108.05620/) (2021)
- [Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing](https://mlsanity.com/paper/2201.12428/) (2022)

## Abstract (arXiv)

Machine learning models are increasingly used in practice. However, many machine learning methods are sensitive to test or operational data that is dissimilar to training data. Out-of-distribution (OOD) data is known to increase the probability of error and research into metrics that identify what dissimilarities in data affect model performance is on-going. Recently, combinatorial coverage metrics have been explored in the literature as an alternative to distribution-based metrics. Results show that coverage metrics can correlate with classification error. However, other results show that the utility of coverage metrics is highly dataset-dependent. In this paper, we show that this dataset-dependence can be alleviated with metric learning, a machine learning technique for learning latent spaces where data from different classes is further apart. In a study of 6 open-source datasets, we find that metric learning increased the difference between set-difference coverage metrics (SDCCMs) calculated on correctly and incorrectly classified data, thereby demonstrating that metric learning improves the ability of SDCCMs to anticipate classification error. Paired t-tests validate the statistical significance of our findings. Overall, we conclude that metric learning improves the ability of coverage metrics to anticipate classifier error and identify when OOD data is likely to degrade model performance.