# Representation Based Complexity Measures for Predicting Generalization in Deep Learning

arXiv: https://arxiv.org/abs/2012.02775 · HTML: https://mlsanity.com/paper/2012.02775/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2020.

## Built on by (most cited first, 29 of 29)

- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [GBK-GNN: Gated Bi-Kernel Graph Neural Networks for Modeling Both Homophily and Heterophily](https://mlsanity.com/paper/2110.15777/) (2021)
- [Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density](https://mlsanity.com/paper/2108.10860/) (2021)
- [Measuring Generalization with Optimal Transport](https://mlsanity.com/paper/2106.03314/) (2021)
- [Homophily-oriented Heterogeneous Graph Rewiring](https://mlsanity.com/paper/2302.06299/) (2023)
- [Predicting Deep Neural Network Generalization with Perturbation Response Curves](https://mlsanity.com/paper/2106.04765/) (2021)
- [MM-GNN: Mix-Moment Graph Neural Network towards Modeling Neighborhood Feature Distribution](https://mlsanity.com/paper/2208.07012/) (2022)
- [Predicting the generalization gap in neural networks using topological data analysis](https://mlsanity.com/paper/2203.12330/) (2022)
- [Revisiting Self-Supervised Heterogeneous Graph Learning from Spectral Clustering Perspective](https://mlsanity.com/paper/2412.00742/) (2024)
- [On Predicting Generalization using GANs](https://mlsanity.com/paper/2111.14212/) (2021)
- [Weight Expansion: A New Perspective on Dropout and Generalization](https://mlsanity.com/paper/2201.09209/) (2022)
- [Pretraining with Random Noise for Fast and Robust Learning without Weight Transport](https://mlsanity.com/paper/2405.16731/) (2024)
- [Neuron with Steady Response Leads to Better Generalization](https://mlsanity.com/paper/2111.15414/) (2021)
- [Input margins can predict generalization too](https://mlsanity.com/paper/2308.15466/) (2023)
- [Position: Cracking the Code of Cascading Disparity Towards Marginalized Communities](https://mlsanity.com/paper/2406.01757/) (2024)
- [Slicing Mutual Information Generalization Bounds for Neural Networks](https://mlsanity.com/paper/2406.04047/) (2024)
- [Enhancing the Influence of Labels on Unlabeled Nodes in Graph Convolutional Networks](https://mlsanity.com/paper/2411.02279/) (2024)
- [Gi and Pal Scores: Deep Neural Network Generalization Statistics](https://mlsanity.com/paper/2104.03469/) (2021)
- [In Search of Probeable Generalization Measures](https://mlsanity.com/paper/2110.12259/) (2021)
- [On the Effects of Artificial Data Modification](https://mlsanity.com/paper/2110.13968/) (2021)
- [A Statistical Model for Predicting Generalization in Few-Shot Classification](https://mlsanity.com/paper/2212.06461/) (2022)
- [The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs](https://mlsanity.com/paper/2302.06925/) (2023)
- [Fantastic DNN Classifiers and How to Identify them without Data](https://mlsanity.com/paper/2305.15563/) (2023)
- [Exploring Dataset-Scale Indicators of Data Quality](https://mlsanity.com/paper/2311.04016/) (2023)
- [A practical generalization metric for deep networks benchmarking](https://mlsanity.com/paper/2409.01498/) (2024)
- [Explaining generalization in deep learning: progress and fundamental limits](https://mlsanity.com/paper/2110.08922/) (2021)
- [On the benefits of non-linear weight updates](https://mlsanity.com/paper/2207.12505/) (2022)
- [Is network fragmentation a useful complexity measure?](https://mlsanity.com/paper/2411.04695/) (2024)
- [Transductive Generalization via Optimal Transport and Its Application to Graph Node Classification](https://mlsanity.com/paper/2603.09257/) (2026)

## Abstract (arXiv)

Deep Neural Networks can generalize despite being significantly overparametrized. Recent research has tried to examine this phenomenon from various view points and to provide bounds on the generalization error or measures predictive of the generalization gap based on these viewpoints, such as norm-based, PAC-Bayes based, and margin-based analysis. In this work, we provide an interpretation of generalization from the perspective of quality of internal representations of deep neural networks, based on neuroscientific theories of how the human visual system creates invariant and untangled object representations. Instead of providing theoretical bounds, we demonstrate practical complexity measures which can be computed ad-hoc to uncover generalization behaviour in deep models. We also provide a detailed description of our solution that won the NeurIPS competition on Predicting Generalization in Deep Learning held at NeurIPS 2020. An implementation of our solution is available at https://github.com/parthnatekar/pgdl.