# Fantastic DNN Classifiers and How to Identify them without Data

arXiv: https://arxiv.org/abs/2305.15563 · HTML: https://mlsanity.com/paper/2305.15563/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Adversarially Robust Generalization Requires More Data](https://mlsanity.com/paper/1804.11285/) (2018)
- [Adversarial Attacks and Defences: A Survey](https://mlsanity.com/paper/1810.00069/) (2018)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Understanding the Failure Modes of Out-of-Distribution Generalization](https://mlsanity.com/paper/2010.15775/) (2020)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [Reconstructing Training Data from Trained Neural Networks](https://mlsanity.com/paper/2206.07758/) (2022)
- [RepMet: Representative-based metric learning for classification and one-shot object detection](https://mlsanity.com/paper/1806.04728/) (2018)
- [Adversarial Defense by Restricting the Hidden Space of Deep Neural Networks](https://mlsanity.com/paper/1904.00887/) (2019)
- [NeurIPS 2020 Competition: Predicting Generalization in Deep Learning](https://mlsanity.com/paper/2012.07976/) (2020)
- [Metric Learning with Adaptive Density Discrimination](https://mlsanity.com/paper/1511.05939/) (2015)
- [Representation Based Complexity Measures for Predicting Generalization in Deep Learning](https://mlsanity.com/paper/2012.02775/) (2020)
- [Predicting Deep Neural Network Generalization with Perturbation Response Curves](https://mlsanity.com/paper/2106.04765/) (2021)
- [Reconstructing Training Data from Multiclass Neural Networks](https://mlsanity.com/paper/2305.03350/) (2023)

## Built on by (most cited first, 1 of 1)

- [Novel Deep Neural Network Classifier Characterization Metrics with Applications to Dataless Evaluation](https://mlsanity.com/paper/2407.13000/) (2024)

## Abstract (arXiv)

Current algorithms and architecture can create excellent DNN classifier models from example data. In general, larger training datasets result in better model estimations, which improve test performance. Existing methods for predicting generalization performance are based on hold-out test examples. To the best of our knowledge, at present no method exists that can estimate the quality of a trained DNN classifier without test data. In this paper, we show that the quality of a trained DNN classifier can be assessed without any example data. We consider DNNs to be composed of a feature extractor and a feature classifier; the feature extractor's output is fed to the classifier. The proposed method iteratively creates class prototypes in the input space for each class by minimizing a cross-entropy loss function at the output of the network. We use these prototypes and their feature relationships to reveal the quality of the classifier. We have developed two metrics: one using the features of the prototypes and the other using adversarial examples corresponding to each prototype. Empirical evaluations show that accuracy obtained from test examples is directly proportional to quality measures obtained from the proposed metrics. We report our observations for ResNet18 with Tiny ImageNet, CIFAR100, and CIFAR10 datasets. The proposed metrics can be used to compare performances of two or more classifiers without test examples.