# A theoretical perspective on mode collapse in variational inference arXiv: https://arxiv.org/abs/2410.13300 ยท HTML: https://mlsanity.com/paper/2410.13300/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 47 references that are in the map. - First posted on arXiv 17 Oct 2024. ## Builds on (most cited first, 27 of 27) - [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015) - [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016) - [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019) - [Flow-based generative models for Markov chain Monte Carlo in lattice field theory](https://mlsanity.com/paper/1904.12072/) (2019) - [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019) - [Solving Statistical Mechanics Using Variational Autoregressive Networks](https://mlsanity.com/paper/1809.10606/) (2018) - [Neural Autoregressive Distribution Estimation](https://mlsanity.com/paper/1605.02226/) (2016) - [High-dimensional limit theorems for SGD: Effective dynamics and critical scaling](https://mlsanity.com/paper/2206.04030/) (2022) - [Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed](https://mlsanity.com/paper/2102.11742/) (2021) - [Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling](https://mlsanity.com/paper/2406.07423/) (2024) - [Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks](https://mlsanity.com/paper/2202.00293/) (2022) - [From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks](https://mlsanity.com/paper/2302.05882/) (2023) - [Variational Boosting: Iteratively Refining Posterior Approximations](https://mlsanity.com/paper/1611.06585/) (2016) - [Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequences](https://mlsanity.com/paper/1609.00978/) (2016) - [Global analysis of Expectation Maximization for mixtures of two Gaussians](https://mlsanity.com/paper/1608.07630/) (2016) - [Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence](https://mlsanity.com/paper/2106.15980/) (2021) - [Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models](https://mlsanity.com/paper/2308.08977/) (2023) - [Detecting and Mitigating Mode-Collapse for Flow-based Sampling of Lattice Field Theories](https://mlsanity.com/paper/2302.14082/) (2023) - [Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD](https://mlsanity.com/paper/2305.18502/) (2023) - [Optimal Protocols for Continual Learning via Statistical Physics and Control Theory](https://mlsanity.com/paper/2409.18061/) (2024) - [Alpha-Beta Divergence For Variational Inference](https://mlsanity.com/paper/1805.01045/) (2018) - [Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians](https://mlsanity.com/paper/2406.04012/) (2024) - [Machine Learning of Thermodynamic Observables in the Presence of Mode Collapse](https://mlsanity.com/paper/2111.11303/) (2021) - [Likelihood landscape and maximum likelihood estimation for the discrete orbit recovery model](https://mlsanity.com/paper/2004.00041/) (2020) - [Local Minima Structures in Gaussian Mixture Models](https://mlsanity.com/paper/2009.13040/) (2020) - [Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training](https://mlsanity.com/paper/2405.18296/) (2024) - [The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions](https://mlsanity.com/paper/2306.10404/) (2023) ## Built on by (most cited first, 14 of 14) - [Temperature-Annealed Boltzmann Generators](https://mlsanity.com/paper/2501.19077/) (2025) - [A solvable model of learning generative diffusion: theory and insights](https://mlsanity.com/paper/2501.03937/) (2025) - [Improving the evaluation of samplers on multi-modal targets](https://mlsanity.com/paper/2504.08916/) (2025) - [Learning Boltzmann Generators via Constrained Mass Transport](https://mlsanity.com/paper/2510.18460/) (2025) - [Variational Inference with Mixtures of Isotropic Gaussians](https://mlsanity.com/paper/2506.13613/) (2025) - [Annealing in variational inference mitigates mode collapse: A theoretical study on Gaussian mixtures](https://mlsanity.com/paper/2602.12923/) (2026) - [FlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Tempering](https://mlsanity.com/paper/2505.10466/) (2025) - [SESaMo: Symmetry-Enforcing Stochastic Modulation for Normalizing Flows](https://mlsanity.com/paper/2505.19619/) (2025) - [When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models](https://mlsanity.com/paper/2602.02855/) (2026) - [The critical slowing down in training diffusion models](https://mlsanity.com/paper/2605.12597/) (2026) - [FrOGS: Discrete Neural Sampler for Independent Alloy Configurations Across Chemical Conditions](https://mlsanity.com/paper/2609.02948/) (2026) - [Mode Collapse of Mean-Field Variational Inference](https://mlsanity.com/paper/2510.17063/) (2025) - [Total Variation Guarantees for Sampling with Stochastic Localization](https://mlsanity.com/paper/2603.29555/) (2026) - [Learning to Predict Distributions over Weight Updates for Test-Time Adaptation](https://mlsanity.com/paper/2610.01934/) (2026) ## Abstract (arXiv) While deep learning has expanded the possibilities for highly expressive variational families, the practical benefits of these tools for variational inference (VI) are often limited by the minimization of the traditional Kullback-Leibler objective, which can yield suboptimal solutions. A major challenge in this context is mode collapse: the phenomenon where a model concentrates on a few modes of the target distribution during training, despite being statistically capable of expressing them all. In this work, we carry a theoretical investigation of mode collapse for the gradient flow on Gaussian mixture models. We identify the key low-dimensional statistics characterizing the flow, and derive a closed set of low-dimensional equations governing their evolution. Leveraging this compact description, we show that mode collapse is present even in statistically favorable scenarios, and identify two key mechanisms driving it: mean alignment and vanishing weight. Our theoretical findings are consistent with the implementation of VI using normalizing flows, a class of popular generative models, thereby offering practical insights.