# Fitting large mixture models using stochastic component selection

arXiv: https://arxiv.org/abs/2110.04776 · HTML: https://mlsanity.com/paper/2110.04776/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 62 references that are in the map.
- First posted on arXiv 10 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [Deep Gaussian Mixture Models](https://mlsanity.com/paper/1711.06929/) (2017)
- [Clustering Millions of Faces by Identity](https://mlsanity.com/paper/1604.00989/) (2016)
- [Deep Transformation-Invariant Clustering](https://mlsanity.com/paper/2006.11132/) (2020)
- [A Unified Approach for Learning the Parameters of Sum-Product Networks](https://mlsanity.com/paper/1601.00318/) (2016)
- [Variational Mixture of Normalizing Flows](https://mlsanity.com/paper/2009.00585/) (2020)
- [Sketching for Large-Scale Learning of Mixture Models](https://mlsanity.com/paper/1606.02838/) (2016)
- [Sketching Datasets for Large-Scale Learning (long version)](https://mlsanity.com/paper/2008.01839/) (2020)
- [Properties of the Stochastic Approximation EM Algorithm with Mini-batch Sampling](https://mlsanity.com/paper/1907.09164/) (2019)
- [Sum-Product-Transform Networks: Exploiting Symmetries using Invertible Transformations](https://mlsanity.com/paper/2005.01297/) (2020)
- [Adaptive Seeding for Gaussian Mixture Models](https://mlsanity.com/paper/1312.5946/) (2013)
- [Fast Learning of Clusters and Topics via Sparse Posteriors](https://mlsanity.com/paper/1609.07521/) (2016)
- [Can clustering scale sublinearly with its clusters? A variational EM acceleration of GMMs and k-means](https://mlsanity.com/paper/1711.03431/) (2017)

## Abstract (arXiv)

Traditional methods for unsupervised learning of finite mixture models require to evaluate the likelihood of all components of the mixture. This becomes computationally prohibitive when the number of components is large, as it is, for example, in the sum-product (transform) networks. Therefore, we propose to apply a combination of the expectation maximization and the Metropolis-Hastings algorithm to evaluate only a small number of, stochastically sampled, components, thus substantially reducing the computational cost. The Markov chain of component assignments is sequentially generated across the algorithm's iterations, having a non-stationary target distribution whose parameters vary via a gradient-descent scheme. We put emphasis on generality of our method, equipping it with the ability to train both shallow and deep mixture models which involve complex, and possibly nonlinear, transformations. The performance of our method is illustrated in a variety of synthetic and real-data contexts, considering deep models, such as mixtures of normalizing flows and sum-product (transform) networks.