# Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Models

arXiv: https://arxiv.org/abs/2103.04333 · HTML: https://mlsanity.com/paper/2103.04333/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 7 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [DeepMutation: Mutation Testing of Deep Learning Systems](https://mlsanity.com/paper/1805.05206/) (2018)
- [DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems](https://mlsanity.com/paper/1812.10812/) (2018)
- [Model-Reuse Attacks on Deep Learning Systems](https://mlsanity.com/paper/1812.00483/) (2018)
- [Automatic Testing and Improvement of Machine Translation](https://mlsanity.com/paper/1910.02688/) (2019)
- [Importance-Driven Deep Learning System Testing](https://mlsanity.com/paper/2002.03433/) (2020)
- [Boosting Operational DNN Testing Efficiency through Conditioning](https://mlsanity.com/paper/1906.02533/) (2019)
- [Scaling Deep Learning on GPU and Knights Landing clusters](https://mlsanity.com/paper/1708.02983/) (2017)
- [Connecting Software Metrics across Versions to Predict Defects](https://mlsanity.com/paper/1712.09835/) (2017)

## Built on by (most cited first, 6 of 6)

- [NeuronFair: Interpretable White-Box Fairness Testing through Biased Neuron Identification](https://mlsanity.com/paper/2112.13214/) (2021)
- [Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities](https://mlsanity.com/paper/2212.10017/) (2022)
- [Reusing Deep Neural Network Models through Model Re-engineering](https://mlsanity.com/paper/2304.00245/) (2023)
- [Modularizing while Training: A New Paradigm for Modularizing DNN Models](https://mlsanity.com/paper/2306.09376/) (2023)
- [Robust Active Learning: Sample-Efficient Training of Robust Deep Learning Models](https://mlsanity.com/paper/2112.02542/) (2021)
- [LaF: Labeling-Free Model Selection for Automated Deep Neural Network Reusing](https://mlsanity.com/paper/2204.03994/) (2022)

## Abstract (arXiv)

The boom of DL technology leads to massive DL models built and shared, which facilitates the acquisition and reuse of DL models. For a given task, we encounter multiple DL models available with the same functionality, which are considered as candidates to achieve this task. Testers are expected to compare multiple DL models and select the more suitable ones w.r.t. the whole testing context. Due to the limitation of labeling effort, testers aim to select an efficient subset of samples to make an as precise rank estimation as possible for these models. To tackle this problem, we propose Sample Discrimination based Selection (SDS) to select efficient samples that could discriminate multiple models, i.e., the prediction behaviors (right/wrong) of these samples would be helpful to indicate the trend of model performance. To evaluate SDS, we conduct an extensive empirical study with three widely-used image datasets and 80 real world DL models. The experimental results show that, compared with state-of-the-art baseline methods, SDS is an effective and efficient sample selection method to rank multiple DL models.