# Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data

arXiv: https://arxiv.org/abs/2002.06716 · HTML: https://mlsanity.com/paper/2002.06716/

## Key facts

- Cited by 115 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2020.

## Built on by (most cited first, 30 of 115)

- [Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights](https://mlsanity.com/paper/2209.14733/) (2022)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [Talking Heads: Understanding Inter-layer Communication in Transformer Language Models](https://mlsanity.com/paper/2406.09519/) (2024)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices](https://mlsanity.com/paper/2306.14263/) (2023)
- [AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models](https://mlsanity.com/paper/2410.10912/) (2024)
- [Taxonomizing local versus global structure in neural network loss landscapes](https://mlsanity.com/paper/2107.11228/) (2021)
- [Hessian Eigenspectra of More Realistic Nonlinear Models](https://mlsanity.com/paper/2103.01519/) (2021)
- [Spectral Evolution and Invariance in Linear-width Neural Networks](https://mlsanity.com/paper/2211.06506/) (2022)
- [Learning continuous models for continuous physics](https://mlsanity.com/paper/2202.08494/) (2022)
- [Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers](https://mlsanity.com/paper/2108.00781/) (2021)
- [Model scale versus domain knowledge in statistical forecasting of chaotic systems](https://mlsanity.com/paper/2303.08011/) (2023)
- [The Universal Weight Subspace Hypothesis](https://mlsanity.com/paper/2512.05117/) (2025)
- [Random matrix analysis of deep neural network weight matrices](https://mlsanity.com/paper/2203.14661/) (2022)
- [OpenMixup: Open Mixup Toolbox and Benchmark for Visual Representation Learning](https://mlsanity.com/paper/2209.04851/) (2022)
- [Multi-Object Navigation with dynamically learned neural implicit representations](https://mlsanity.com/paper/2210.05129/) (2022)
- [Small Singular Values Matter: A Random Matrix Analysis of Transformer Models](https://mlsanity.com/paper/2410.17770/) (2024)
- [Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training](https://mlsanity.com/paper/2312.00359/) (2023)
- [AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality](https://mlsanity.com/paper/2410.10054/) (2024)
- [Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime](https://mlsanity.com/paper/2509.24882/) (2025)
- [Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics](https://mlsanity.com/paper/2106.00734/) (2021)
- [Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data](https://mlsanity.com/paper/2202.02842/) (2022)
- [An Operator Theoretic View on Pruning Deep Neural Networks](https://mlsanity.com/paper/2110.14856/) (2021)
- [Investigating Power laws in Deep Representation Learning](https://mlsanity.com/paper/2202.05808/) (2022)
- [A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities](https://mlsanity.com/paper/2410.18938/) (2024)
- [A Three-regime Model of Network Pruning](https://mlsanity.com/paper/2305.18383/) (2023)
- [An overview of model uncertainty and variability in LLM-based sentiment analysis. Challenges, mitigation strategies and the role of explainability](https://mlsanity.com/paper/2504.04462/) (2025)
- [AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs](https://mlsanity.com/paper/2506.14562/) (2025)
- [Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping](https://mlsanity.com/paper/2111.13331/) (2021)
- [Hyper-Representations for Pre-Training and Transfer Learning](https://mlsanity.com/paper/2207.10951/) (2022)

## Abstract (arXiv)

In many applications, one works with neural network models trained by someone else. For such pretrained models, one may not have access to training data or test data. Moreover, one may not know details about the model, e.g., the specifics of the training data, the loss function, the hyperparameter values, etc. Given one or many pretrained models, it is a challenge to say anything about the expected performance or quality of the models. Here, we address this challenge by providing a detailed meta-analysis of hundreds of publicly-available pretrained models. We examine norm based capacity control metrics as well as power law based metrics from the recently-developed Theory of Heavy-Tailed Self Regularization. We find that norm based metrics correlate well with reported test accuracies for well-trained models, but that they often cannot distinguish well-trained versus poorly-trained models. We also find that power law based metrics can do much better -- quantitatively better at discriminating among series of well-trained models with a given architecture; and qualitatively better at discriminating well-trained versus poorly-trained models. These methods can be used to identify when a pretrained neural network has problems that cannot be detected simply by examining training/test accuracies.