Fantastic DNN Classifiers and How to Identify them without Data
Not on the map. Why?
Abstract (arXiv)
Current algorithms and architecture can create excellent DNN classifier models from example data. In general, larger training datasets result in better model estimations, which improve test performance. Existing methods for predicting generalization performance are based on hold-out test examples. To the best of our knowledge, at present no method exists that can estimate the quality of a trained DNN classifier without test data. In this paper, we show that the quality of a trained DNN classifier can be assessed without any example data. We consider DNNs to be composed of a feature extractor and a feature classifier; the feature extractor's output is fed to the classifier. The proposed method iteratively creates class prototypes in the input space for each class by minimizing a cross-entropy loss function at the output of the network. We use these prototypes and their feature relationships to reveal the quality of the classifier. We have developed two metrics: one using the features of the prototypes and the other using adversarial examples corresponding to each prototype. Empirical evaluations show that accuracy obtained from test examples is directly proportional to quality measures obtained from the proposed metrics. We report our observations for ResNet18 with Tiny ImageNet, CIFAR100, and CIFAR10 datasets. The proposed metrics can be used to compare performances of two or more classifiers without test examples.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 24 May 2023.
Builds on
20 of 26 references.
References · 20
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2017Prototypical Networks for Few-shot Learningcited by 3,015
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
- 2018Robustness May Be at Odds with Accuracycited by 880
- 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
- 2018Adversarially Robust Generalization Requires More Datacited by 318
- 2018Adversarial Attacks and Defences: A Surveycited by 259
- 2017Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictionscited by 234
- 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
- 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
- 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 141
8 more
- 2022Reconstructing Training Data from Trained Neural Networkscited by 133
- 2018RepMet: Representative-based metric learning for classification and one-shot object detectioncited by 85
- 2019Adversarial Defense by Restricting the Hidden Space of Deep Neural Networkscited by 56
- 2020NeurIPS 2020 Competition: Predicting Generalization in Deep Learningcited by 41
- 2015Metric Learning with Adaptive Density Discriminationcited by 40
- 2020Representation Based Complexity Measures for Predicting Generalization in Deep Learningcited by 29
- 2021Predicting Deep Neural Network Generalization with Perturbation Response Curvescited by 14
- 2023Reconstructing Training Data from Multiclass Neural Networkscited by 7
Built on by
Citing papers per year: 2024: 1.
1 paper.