# Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Models

arXiv: https://arxiv.org/abs/2202.08786 · HTML: https://mlsanity.com/paper/2202.08786/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 41 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 7 of 7)

- [Statistical guarantees for the EM algorithm: From population to sample-based analysis](https://mlsanity.com/paper/1408.2156/) (2014)
- [Singularity, Misspecification, and the Convergence Rate of EM](https://mlsanity.com/paper/1810.00828/) (2018)
- [Tight bounds for learning a mixture of two gaussians](https://mlsanity.com/paper/1404.4997/) (2014)
- [Global Convergence of EM Algorithm for Mixtures of Two Component Linear Regression](https://mlsanity.com/paper/1810.05752/) (2018)
- [Sharp Analysis of Expectation-Maximization for Weakly Identifiable Models](https://mlsanity.com/paper/1902.00194/) (2019)
- [Estimating the Number of Components in Finite Mixture Models via the Group-Sort-Fuse Procedure](https://mlsanity.com/paper/2005.11641/) (2020)
- [Uniform Convergence Rates for Maximum Likelihood Estimation under Two-Component Gaussian Mixture Models](https://mlsanity.com/paper/2006.00704/) (2020)

## Built on by (most cited first, 22 of 22)

- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [Mixture of Experts Meets Prompt-Based Continual Learning](https://mlsanity.com/paper/2405.14124/) (2024)
- [Demystifying Softmax Gating Function in Gaussian Mixture of Experts](https://mlsanity.com/paper/2305.03288/) (2023)
- [Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2309.13850/) (2023)
- [Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts](https://mlsanity.com/paper/2405.13997/) (2024)
- [A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts](https://mlsanity.com/paper/2310.14188/) (2023)
- [On Least Square Estimation in Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2402.02952/) (2024)
- [Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts](https://mlsanity.com/paper/2305.07572/) (2023)
- [Statistical Advantages of Perturbing Cosine Router in Mixture of Experts](https://mlsanity.com/paper/2405.14131/) (2024)
- [Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?](https://mlsanity.com/paper/2401.13875/) (2024)
- [Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts](https://mlsanity.com/paper/2410.02200/) (2024)
- [RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts](https://mlsanity.com/paper/2502.03044/) (2025)
- [Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts](https://mlsanity.com/paper/2501.18936/) (2025)
- [Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective](https://mlsanity.com/paper/2502.00281/) (2025)
- [On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation](https://mlsanity.com/paper/2502.03029/) (2025)
- [DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks](https://mlsanity.com/paper/2510.04331/) (2025)
- [Dendrogram of mixing measures: Hierarchical clustering and model selection for finite mixture models](https://mlsanity.com/paper/2403.01684/) (2024)
- [Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads](https://mlsanity.com/paper/2510.04295/) (2025)
- [A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts](https://mlsanity.com/paper/2602.01468/) (2026)
- [Characterizing Heterogeneous Rates in Finite Mixture Estimation via Partial Optimal Transport](https://mlsanity.com/paper/2609.16622/) (2026)
- [Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function](https://mlsanity.com/paper/2602.01466/) (2026)
- [Fidelity-Diversity Metrics for Text](https://mlsanity.com/paper/2607.04563/) (2026)

## Abstract (arXiv)

We revisit the classical problem of deriving convergence rates for the maximum likelihood estimator (MLE) in finite mixture models. The Wasserstein distance has become a standard loss function for the analysis of parameter estimation in these models, due in part to its ability to circumvent label switching and to accurately characterize the behaviour of fitted mixture components with vanishing weights. However, the Wasserstein distance is only able to capture the worst-case convergence rate among the remaining fitted mixture components. We demonstrate that when the log-likelihood function is penalized to discourage vanishing mixing weights, stronger loss functions can be derived to resolve this shortcoming of the Wasserstein distance. These new loss functions accurately capture the heterogeneity in convergence rates of fitted mixture components, and we use them to sharpen existing pointwise and uniform convergence rates in various classes of mixture models. In particular, these results imply that a subset of the components of the penalized MLE typically converge significantly faster than could have been anticipated from past work. We further show that some of these conclusions extend to the traditional MLE. Our theoretical findings are supported by a simulation study to illustrate these improved convergence rates.