# On Predicting Generalization using GANs

arXiv: https://arxiv.org/abs/2111.14212 · HTML: https://mlsanity.com/paper/2111.14212/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 28 Nov 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks](https://mlsanity.com/paper/1805.12076/) (2018)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Predicting the Generalization Gap in Deep Networks with Margin Distributions](https://mlsanity.com/paper/1810.00113/) (2018)
- [In Search of Robust Measures of Generalization](https://mlsanity.com/paper/2010.11924/) (2020)
- [Detecting Overfitting of Deep Generative Networks via Latent Recovery](https://mlsanity.com/paper/1901.03396/) (2019)
- [NeurIPS 2020 Competition: Predicting Generalization in Deep Learning](https://mlsanity.com/paper/2012.07976/) (2020)
- [Generalization bounds via distillation](https://mlsanity.com/paper/2104.05641/) (2021)
- [Representation Based Complexity Measures for Predicting Generalization in Deep Learning](https://mlsanity.com/paper/2012.02775/) (2020)
- [Robustness to Augmentations as a Generalization metric](https://mlsanity.com/paper/2101.06459/) (2021)
- [Ranking Deep Learning Generalization using Label Variation in Latent Geometry Graphs](https://mlsanity.com/paper/2011.12737/) (2020)

## Built on by (most cited first, 6 of 6)

- [PAC-Bayesian Generalization Bounds for Adversarial Generative Models](https://mlsanity.com/paper/2302.08942/) (2023)
- [FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognition](https://mlsanity.com/paper/2305.17421/) (2023)
- [Intra- & Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization](https://mlsanity.com/paper/2307.00648/) (2023)
- [CAME: Contrastive Automated Model Evaluation](https://mlsanity.com/paper/2308.11111/) (2023)
- [Input margins can predict generalization too](https://mlsanity.com/paper/2308.15466/) (2023)
- [Using Synthetic Data to estimate the True Error is theoretically and practically doable](https://mlsanity.com/paper/2511.00964/) (2025)

## Abstract (arXiv)

Research on generalization bounds for deep networks seeks to give ways to predict test error using just the training dataset and the network parameters. While generalization bounds can give many insights about architecture design, training algorithms, etc., what they do not currently do is yield good predictions for actual test error. A recently introduced Predicting Generalization in Deep Learning competition~citepjiang2020neurips aims to encourage discovery of methods to better predict test error. The current paper investigates a simple idea: can test error be predicted using em synthetic data, produced using a Generative Adversarial Network (GAN) that was trained on the same training dataset? Upon investigating several GAN models and architectures, we find that this turns out to be the case. In fact, using GANs pre-trained on standard datasets, the test error can be predicted without requiring any additional hyper-parameter tuning. This result is surprising because GANs have well-known limitations (e.g. mode collapse) and are known to not learn the data distribution accurately. Yet the generated samples are good enough to substitute for test data. Several additional experiments are presented to explore reasons why GANs do well at this task. In addition to a new approach for predicting generalization, the counter-intuitive phenomena presented in our work may also call for a better understanding of GANs' strengths and limitations.