mlsanityData to 14 Oct 2026

A theoretical perspective on mode collapse in variational inference

arXiv 2410.13300 ↗First posted 17 Oct 2024stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 14

Not on the map. Why?

Abstract (arXiv)

While deep learning has expanded the possibilities for highly expressive variational families, the practical benefits of these tools for variational inference (VI) are often limited by the minimization of the traditional Kullback-Leibler objective, which can yield suboptimal solutions. A major challenge in this context is mode collapse: the phenomenon where a model concentrates on a few modes of the target distribution during training, despite being statistically capable of expressing them all. In this work, we carry a theoretical investigation of mode collapse for the gradient flow on Gaussian mixture models. We identify the key low-dimensional statistics characterizing the flow, and derive a closed set of low-dimensional equations governing their evolution. Leveraging this compact description, we show that mode collapse is present even in statistically favorable scenarios, and identify two key mechanisms driving it: mean alignment and vanishing weight. Our theoretical findings are consistent with the implementation of VI using normalizing flows, a class of popular generative models, thereby offering practical insights.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

27 of 47 references.

References · 27

  1. 2015Variational Inference with Normalizing Flowscited by 2,170
  2. 2016Density estimation using Real NVPcited by 2,042
  3. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  4. 2019Flow-based generative models for Markov chain Monte Carlo in lattice field theorycited by 148
  5. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  6. 2018Solving Statistical Mechanics Using Variational Autoregressive Networkscited by 100
  7. 2016Neural Autoregressive Distribution Estimationcited by 95
  8. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 70
  9. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  10. 2024Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Samplingcited by 56
  11. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  12. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 43
15 more
  1. 2016Variational Boosting: Iteratively Refining Posterior Approximationscited by 37
  2. 2016Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequencescited by 34
  3. 2016Global analysis of Expectation Maximization for mixtures of two Gaussianscited by 31
  4. 2021Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergencecited by 29
  5. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 26
  6. 2023Detecting and Mitigating Mode-Collapse for Flow-based Sampling of Lattice Field Theoriescited by 25
  7. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 15
  8. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  9. 2018Alpha-Beta Divergence For Variational Inferencecited by 12
  10. 2024Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussianscited by 11
  11. 2021Machine Learning of Thermodynamic Observables in the Presence of Mode Collapsecited by 10
  12. 2020Likelihood landscape and maximum likelihood estimation for the discrete orbit recovery modelcited by 9
  13. 2020Local Minima Structures in Gaussian Mixture Modelscited by 9
  14. 2024Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Trainingcited by 8
  15. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025Temperature-Annealed Boltzmann Generatorscited by 15
  2. 2025A solvable model of learning generative diffusion: theory and insightscited by 14
  3. 2025Improving the evaluation of samplers on multi-modal targetscited by 9
  4. 2025Learning Boltzmann Generators via Constrained Mass Transportcited by 9
  5. 2025Variational Inference with Mixtures of Isotropic Gaussianscited by 6
  6. 2026Annealing in variational inference mitigates mode collapse: A theoretical study on Gaussian mixturescited by 4
  7. 2025FlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Temperingcited by 2
  8. 2025SESaMo: Symmetry-Enforcing Stochastic Modulation for Normalizing Flowscited by 2
  9. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  10. 2026The critical slowing down in training diffusion modelscited by 1
  11. 2026FrOGS: Discrete Neural Sampler for Independent Alloy Configurations Across Chemical Conditionscited by 1
  12. 2025Mode Collapse of Mean-Field Variational Inferencecited by 0
2 more
  1. 2026Total Variation Guarantees for Sampling with Stochastic Localizationcited by 0
  2. 2026Learning to Predict Distributions over Weight Updates for Test-Time Adaptationcited by 0