mlsanityData to 7 Oct 2026

Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions

arXiv 2106.03791 ↗First posted 7 Jun 2021stat.ML cond-mat.dis-nn cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

Generalised linear models for multi-class classification problems are one of the fundamental building blocks of modern machine learning tasks. In this manuscript, we characterise the learning of a mixture of K Gaussians with generic means and covariances via empirical risk minimisation (ERM) with any convex loss and regularisation. In particular, we prove exact asymptotics characterising the ERM estimator in high-dimensions, extending several previous results about Gaussian mixture classification in the literature. We exemplify our result in two tasks of interest in statistical learning: a) classification for a mixture with sparse means, where we study the efficiency of ℓ1 penalty with respect to ℓ2; b) max-margin multi-class classification, where we characterise the phase transition on the existence of the multi-class logistic maximum likelihood estimator for K>2. Finally, we discuss how our theory can be applied beyond the scope of synthetic data, showing that in different cases Gaussian mixtures capture closely the learning curve of classification tasks in real data sets.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

34 of 69 references.

References · 34

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  8. 2018To understand deep learning we need to understand kernel learningcited by 186
  9. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  10. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  11. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  12. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
22 more
  1. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  2. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  3. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  4. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  5. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  6. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  7. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  8. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  9. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  10. 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 45
  11. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  12. 2020The Lasso with general Gaussian designs with applications to hypothesis testingcited by 40
  13. 2013A framework to characterize performance of LASSO algorithmscited by 38
  14. 2019The Impact of Regularization on High-dimensional Logistic Regressioncited by 35
  15. 2020Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic Viewcited by 32
  16. 2020Analytic Study of Double Descent in Binary Classification: The Impact of Losscited by 28
  17. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  18. 2020Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)cited by 23
  19. 2016Finite Sample Analysis of Approximate Message Passing Algorithmscited by 20
  20. 2020Universality of Approximate Message Passing Algorithmscited by 16
  21. 2017Multi-Layer Generalized Linear Estimationcited by 12
  22. 2017Regularizing with Bregman-Moreau envelopescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 11; 2024: 17; 2025: 12; 2026: 3.

52 papers.

Sort:

By area:

  1. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  2. 2024IBD-PSC: Input-level Backdoor Detection via Parameter-oriented Scaling Consistencycited by 38
  3. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  4. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  5. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  6. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  7. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
  8. 2021Probing transfer learning with a model of synthetic correlated datasetscited by 24
  9. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
  10. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  11. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  12. 2023High-dimensional robust regression under heavy-tailed data: Asymptotics and Universalitycited by 15
38 more
  1. 2024High-dimensional learning of narrow neural networkscited by 12
  2. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  3. 2025Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formulacited by 11
  4. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  5. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  6. 2023Regularized Linear Regression for Binary Classificationcited by 7
  7. 2022Observable adjustments in single-index models for regularized M-estimatorscited by 5
  8. 2024A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offscited by 5
  9. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 5
  10. 2023Asymptotic Bayes risk of semi-supervised multitask learning on Gaussian mixturecited by 4
  11. 2023Least Squares Regression Can Exhibit Under-Parameterized Double Descentcited by 4
  12. 2024A Novel Gaussian Min-Max Theorem and its Applicationscited by 4
  13. 2024Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regressioncited by 4
  14. 2024A replica analysis of under-baggingcited by 4
  15. 2024The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixturescited by 4
  16. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  17. 2023Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensionscited by 3
  18. 2023Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Modelscited by 3
  19. 2023Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networkscited by 3
  20. 2024Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalancecited by 2
  21. 2024Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Settingcited by 2
  22. 2021Optimal regularizations for data generation with probabilistic graphical modelscited by 1
  23. 2022The impact of memory on learning sequence-to-sequence taskscited by 1
  24. 2024A Convergence Analysis of Approximate Message Passing with Non-Separable Functions and Applications to Multi-Class Classificationcited by 1
  25. 2024One-Bit Quantization and Sparsification for Multiclass Linear Classification with Strong Regularizationcited by 1
  26. 2024When resampling/reweighting improves feature learning in imbalanced classification?: A toy-model studycited by 1
  27. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 1
  28. 2025On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classificationcited by 1
  29. 2025Thumb on the Scale: Optimal Loss Weighting in Last Layer Retrainingcited by 1
  30. 2025Breaking the curse of dimensionality for linear rules: optimal predictors over the ellipsoidcited by 1
  31. 2026Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimizationcited by 1
  32. 2022Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learningcited by 0
  33. 2024Classifying Overlapping Gaussian Mixtures in High Dimensions: From Optimal Classifiers to Neural Netscited by 0
  34. 2024Statistical Inference in Classification of High-dimensional Gaussian Mixturecited by 0
  35. 2024Analysis of High-dimensional Gaussian Labeled-unlabeled Mixture Model via Message-passing Algorithmcited by 0
  36. 2025Gaussian Universality for Diffusion Modelscited by 0
  37. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 0
  38. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0

50 most cited of 52 All 52 (JSON)