The SVHN Dataset Is Deceptive for Probabilistic Generative Models Due to a Distribution Mismatch
Not on the map. Why?
Abstract (arXiv)
The Street View House Numbers (SVHN) dataset is a popular benchmark dataset in deep learning. Originally designed for digit classification tasks, the SVHN dataset has been widely used as a benchmark for various other tasks including generative modeling. However, with this work, we aim to warn the community about an issue of the SVHN dataset as a benchmark for generative modeling tasks: we discover that the official split into training set and test set of the SVHN dataset are not drawn from the same distribution. We empirically show that this distribution mismatch has little impact on the classification task (which may explain why this issue has not been detected before), but it severely affects the evaluation of probabilistic generative models, such as Variational Autoencoders and diffusion models. As a workaround, we propose to mix and re-split the official training and test set when SVHN is used for tasks other than classification. We publish a new split and the indices we used to create it at https://jzenn.github.io/svhn-remix/ .
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 30 Oct 2023.
Builds on
23 of 35 references.
References · 23
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2022Generative Adversarial Networkscited by 5,603
- 2014Going Deeper with Convolutionscited by 4,231
- 2016Improved Techniques for Training GANscited by 2,938
- 2015Variational Inference with Normalizing Flowscited by 2,161
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
- 2021Variational Diffusion Modelscited by 1,145
11 more
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
- 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
- 2013Network In Networkcited by 600
- 2018Do Deep Generative Models Know What They Don't Know?cited by 443
- 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
- 2019Input complexity and out-of-distribution detection with likelihood-based generative modelscited by 186
- 2020Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encodercited by 145
- 2018Constructing Unrestricted Adversarial Examples with Generative Modelscited by 135
- 2018Improving GANs Using Optimal Transportcited by 122
- 2016Auxiliary Deep Generative Modelscited by 72
- 2017Adversarial examples for generative modelscited by 58
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Ctrl-A: Control-Driven Online Data Augmentationcited by 0
- 2026Ctrl-A: Control-Driven Online Data Augmentation23 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2024Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections
- 2025On the Statistical Capacity of Deep Generative Models
- 2024An analysis of the noise schedule for score-based generative models
- 2023Understanding Deep Generative Models with Generalized Empirical Likelihoods
- 2021Score-based Generative Modeling in Latent Space
- 2025Efficient Long-Tail Learning in Latent Space by sampling Synthetic Data
- 2023Improving the Effectiveness of Deep Generative Data
- 2022A Study on the Evaluation of Generative Models