# Detecting Overfitting of Deep Generative Networks via Latent Recovery

arXiv: https://arxiv.org/abs/1901.03396 · HTML: https://mlsanity.com/paper/1901.03396/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2019.

## Built on by (most cited first, 30 of 44)

- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Regularizing Generative Adversarial Networks under Limited Data](https://mlsanity.com/paper/2104.03310/) (2021)
- [Visual Prompt Tuning for Generative Transfer Learning](https://mlsanity.com/paper/2210.00990/) (2022)
- [Membership Inference Attacks against Synthetic Data through Overfitting Detection](https://mlsanity.com/paper/2302.12580/) (2023)
- [On Memorization in Probabilistic Deep Generative Models](https://mlsanity.com/paper/2106.03216/) (2021)
- [Labels4Free: Unsupervised Segmentation using StyleGAN](https://mlsanity.com/paper/2103.14968/) (2021)
- [Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data](https://mlsanity.com/paper/2111.06849/) (2021)
- [Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data](https://mlsanity.com/paper/2109.11429/) (2021)
- [When do GANs replicate? On the choice of dataset size](https://mlsanity.com/paper/2202.11765/) (2022)
- [This Person (Probably) Exists. Identity Membership Attacks Against GAN Generated Faces](https://mlsanity.com/paper/2107.06018/) (2021)
- [Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples](https://mlsanity.com/paper/2302.04440/) (2023)
- [Exploring Incompatible Knowledge Transfer in Few-shot Image Generation](https://mlsanity.com/paper/2304.07574/) (2023)
- [On the Privacy Properties of GAN-generated Samples](https://mlsanity.com/paper/2206.01349/) (2022)
- [KD-DLGAN: Data Limited Image Generation via Knowledge Distillation](https://mlsanity.com/paper/2303.17158/) (2023)
- [GenCo: Generative Co-training for Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2110.01254/) (2021)
- [Manifold Learning Benefits GANs](https://mlsanity.com/paper/2112.12618/) (2021)
- [The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasets](https://mlsanity.com/paper/2312.05114/) (2023)
- [Graphical vs. Deep Generative Models: Measuring the Impact of Differentially Private Mechanisms and Budgets on Utility](https://mlsanity.com/paper/2305.10994/) (2023)
- [RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection](https://mlsanity.com/paper/2302.12464/) (2023)
- [medXGAN: Visual Explanations for Medical Classifiers through a Generative Latent Space](https://mlsanity.com/paper/2204.05376/) (2022)
- [A Multi-Modal Contrastive Diffusion Model for Therapeutic Peptide Generation](https://mlsanity.com/paper/2312.15665/) (2023)
- [ZeroDiff: Solidified Visual-Semantic Correlation in Zero-Shot Learning](https://mlsanity.com/paper/2406.02929/) (2024)
- [Generalization Metrics for Practical Quantum Advantage in Generative Models](https://mlsanity.com/paper/2201.08770/) (2022)
- [DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?](https://mlsanity.com/paper/2111.13617/) (2021)
- [DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data](https://mlsanity.com/paper/2211.14694/) (2022)
- [On Predicting Generalization using GANs](https://mlsanity.com/paper/2111.14212/) (2021)
- [Haven't I Seen You Before? Assessing Identity Leakage in Synthetic Irises](https://mlsanity.com/paper/2211.05629/) (2022)
- [An Inversion-based Measure of Memorization for Diffusion Models](https://mlsanity.com/paper/2405.05846/) (2024)
- [Domain adaptation in small-scale and heterogeneous biological datasets](https://mlsanity.com/paper/2405.19221/) (2024)
- [Reducing Training Sample Memorization in GANs by Training with Memorization Rejection](https://mlsanity.com/paper/2210.12231/) (2022)

## Abstract (arXiv)

State of the art deep generative networks are capable of producing images with such incredible realism that they can be suspected of memorizing training images. It is why it is not uncommon to include visualizations of training set nearest neighbors, to suggest generated images are not simply memorized. We demonstrate this is not sufficient and motivates the need to study memorization/overfitting of deep generators with more scrutiny. This paper addresses this question by i) showing how simple losses are highly effective at reconstructing images for deep generators ii) analyzing the statistics of reconstruction errors when reconstructing training and validation images, which is the standard way to analyze overfitting in machine learning. Using this methodology, this paper shows that overfitting is not detectable in the pure GAN models proposed in the literature, in contrast with those using hybrid adversarial losses, which are amongst the most widely applied generative methods. The paper also shows that standard GAN evaluation metrics fail to capture memorization for some deep generators. Finally, the paper also shows how off-the-shelf GAN generators can be successfully applied to face inpainting and face super-resolution using the proposed reconstruction method, without hybrid adversarial losses.