mlsanityData to 7 Oct 2026

A Mathematical Framework for Learning Probability Distributions

arXiv 2212.11481 ↗First posted 22 Dec 2022stat.ML cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 82 of its 134 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

82 of 134 references.

References · 82

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  7. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  8. 2017Improved Training of Wasserstein GANscited by 2,223
  9. 2015Variational Inference with Normalizing Flowscited by 2,161
  10. 2014Conditional Generative Adversarial Netscited by 2,152
  11. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  12. 2016Density estimation using Real NVPcited by 2,029
  13. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  14. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  15. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  16. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  17. 2016Pixel Recurrent Neural Networkscited by 814
  18. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  19. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  20. 2018Which Training Methods for GANs do actually Converge?cited by 672
  21. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  22. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  23. 2017Wasserstein GANcited by 635
  24. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  25. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  26. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  27. 2021GFlowNet Foundationscited by 286
  28. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 266
  29. 2015Generative Moment Matching Networkscited by 252
  30. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  31. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  32. 2020A Review on Generative Adversarial Networks: Algorithms, Theory, and Applicationscited by 209
  33. 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
  34. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  35. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  36. 2019Mode Seeking Generative Adversarial Networks for Diverse Image Synthesiscited by 138
  37. 2016Mode Regularized Generative Adversarial Networkscited by 123
  38. 2016Energy-based Generative Adversarial Networkcited by 119

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 2.

7 papers.

  1. 2023On the Generalization Properties of Diffusion Modelscited by 67
  2. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  3. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 28
  4. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 15
  5. 2025Implicit Regularisation in Diffusion Models: An Algorithm-Dependent Generalisation Analysiscited by 6
  6. 2023Cluster-based classification with neural ODEs via controlcited by 4
  7. 2024Hamiltonian bridge: A physics-driven generative framework for targeted pattern controlcited by 1