# Novel Deep Neural Network Classifier Characterization Metrics with Applications to Dataless Evaluation

arXiv: https://arxiv.org/abs/2407.13000 · HTML: https://mlsanity.com/paper/2407.13000/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Reconstructing Training Data from Trained Neural Networks](https://mlsanity.com/paper/2206.07758/) (2022)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Adversarial Defense by Restricting the Hidden Space of Deep Neural Networks](https://mlsanity.com/paper/1904.00887/) (2019)
- [Metric Learning with Adaptive Density Discrimination](https://mlsanity.com/paper/1511.05939/) (2015)
- [Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation](https://mlsanity.com/paper/2202.06881/) (2022)
- [Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends](https://mlsanity.com/paper/2401.13177/) (2024)
- [Reconstructing Training Data from Multiclass Neural Networks](https://mlsanity.com/paper/2305.03350/) (2023)
- [Learning More Universal Representations for Transfer-Learning](https://mlsanity.com/paper/1712.09708/) (2017)
- [A New Perspective for Understanding Generalization Gap of Deep Neural Networks Trained with Large Batch Sizes](https://mlsanity.com/paper/2210.12184/) (2022)
- [Fantastic DNN Classifiers and How to Identify them without Data](https://mlsanity.com/paper/2305.15563/) (2023)

## Abstract (arXiv)

The mainstream AI community has seen a rise in large-scale open-source classifiers, often pre-trained on vast datasets and tested on standard benchmarks; however, users facing diverse needs and limited, expensive test data may be overwhelmed by available choices. Deep Neural Network (DNN) classifiers undergo training, validation, and testing phases using example dataset, with the testing phase focused on determining the classification accuracy of test examples without delving into the inner working of the classifier. In this work, we evaluate a DNN classifier's training quality without any example dataset. It is assumed that a DNN is a composition of a feature extractor and a classifier which is the penultimate completely connected layer. The quality of a classifier is estimated using its weight vectors. The feature extractor is characterized using two metrics that utilize feature vectors it produces when synthetic data is fed as input. These synthetic input vectors are produced by backpropagating desired outputs of the classifier. Our empirical study of the proposed method for ResNet18, trained with CAFIR10 and CAFIR100 datasets, confirms that data-less evaluation of DNN classifiers is indeed possible.