# Gradient-based training of Gaussian Mixture Models for High-Dimensional Streaming Data

arXiv: https://arxiv.org/abs/1912.09379 · HTML: https://mlsanity.com/paper/1912.09379/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2019.

## Built on by (most cited first, 13 of 13)

- [Tractable Probabilistic Graph Representation Learning with Graph-Induced Sum-Product Networks](https://mlsanity.com/paper/2305.10544/) (2023)
- [Slicing the Gaussian Mixture Wasserstein Distance](https://mlsanity.com/paper/2504.08544/) (2025)
- [A Study of Continual Learning Methods for Q-Learning](https://mlsanity.com/paper/2206.03934/) (2022)
- [Adiabatic replay for continual learning](https://mlsanity.com/paper/2303.13157/) (2023)
- [Class-Incremental Mixture of Gaussians for Deep Continual Learning](https://mlsanity.com/paper/2307.04094/) (2023)
- [This Probably Looks Exactly Like That: An Invertible Prototypical Network](https://mlsanity.com/paper/2407.12200/) (2024)
- [Performance of Gaussian Mixture Model Classifiers on Embedded Feature Spaces](https://mlsanity.com/paper/2410.13421/) (2024)
- [A new perspective on probabilistic image modeling](https://mlsanity.com/paper/2203.11034/) (2022)
- [A differentiable Gaussian Prototype Layer for explainable Segmentation](https://mlsanity.com/paper/2306.14361/) (2023)
- [Large-scale gradient-based training of Mixtures of Factor Analyzers](https://mlsanity.com/paper/2308.13778/) (2023)
- [Hierarchical novel class discovery for single-cell transcriptomic profiles](https://mlsanity.com/paper/2409.05937/) (2024)
- [A Mapper Algorithm with implicit intervals and its optimization](https://mlsanity.com/paper/2412.11631/) (2024)
- [Fair Model-based Clustering](https://mlsanity.com/paper/2602.21509/) (2026)

## Abstract (arXiv)

We present an approach for efficiently training Gaussian Mixture Model (GMM) by Stochastic Gradient Descent (SGD) with non-stationary, high-dimensional streaming data. Our training scheme does not require data-driven parameter initialization (e.g., k-means) and can thus be trained based on a random initialization. Furthermore, the approach allows mini-batch sizes as low as 1, which are typical for streaming-data settings. Major problems in such settings are undesirable local optima during early training phases and numerical instabilities due to high data dimensionalities. We introduce an adaptive annealing procedure to address the first problem, whereas numerical instabilities are eliminated by using an exponential-free approximation to the standard GMM log-likelihood. Experiments on a variety of visual and non-visual benchmarks show that our SGD approach can be trained completely without, for instance, k-means based centroid initialization. It also compares favorably to an online variant of Expectation-Maximization (EM) - stochastic EM (sEM), which it outperforms by a large margin for very high-dimensional data.