mlsanityData to 7 Oct 2026

Universality laws for Gaussian mixtures in generalized linear models

arXiv 2302.08933 ↗First posted 17 Feb 2023math.ST stat.ML stat.THcited by 28

Not on the map. Why?

Abstract (arXiv)

Let (xi, yi)i=1,dots,n denote independent samples from a general mixture distribution sumc∈CρcPcx, and consider the hypothesis class of generalized linear models haty = F(Thetatopx). In this work, we investigate the asymptotic joint statistics of the family of generalized linear estimators (Theta1, dots, ThetaM) obtained either from (a) minimizing an empirical risk hatRn(Theta;X,y) or (b) sampling from the associated Gibbs measure exp(-β n hatRn(Theta;X,y)). Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution Pcx. In particular, this allow us to prove the universality of different quantities of interest, such as the training and generalization errors, redeeming a recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis. Finally, we discuss the applications of our results to different machine learning tasks of interest, such as ensembling and uncertainty

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 33 of its 86 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

33 of 86 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2015Variational Inference with Normalizing Flowscited by 2,161
  6. 2014Conditional Generative Adversarial Netscited by 2,152
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  8. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2019Benign Overfitting in Linear Regressioncited by 544
  11. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  12. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
21 more
  1. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  2. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  3. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  4. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  5. 2017A Random Matrix Approach to Neural Networkscited by 80
  6. 2022Universality of empirical risk minimizationcited by 73
  7. 2020Understanding and Mitigating Exploding Inverses in Invertible Neural Networkscited by 71
  8. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  9. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  10. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  11. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  12. 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 45
  13. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  14. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  15. 2020The Lasso with general Gaussian designs with applications to hypothesis testingcited by 40
  16. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  17. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  18. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
  19. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  20. 2019High Dimensional Classification via Regularized and Unregularized Empirical Risk Minimization: Precise Error and Optimal Losscited by 8
  21. 2019On the Universality of Noiseless Linear Estimation with Respect to the Measurement Matrixcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 8; 2025: 10; 2026: 4.

28 papers.

Sort:

  1. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  2. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  3. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  4. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
  5. 2024High-dimensional learning of narrow neural networkscited by 12
  6. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  7. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  8. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  9. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  10. 2024A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offscited by 5
  11. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 5
  12. 2024A Novel Gaussian Min-Max Theorem and its Applicationscited by 4
16 more
  1. 2024The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixturescited by 4
  2. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
  3. 2023Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensionscited by 3
  4. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  5. 2024On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Boundscited by 2
  6. 2026Deep Learning of Compositional Targets with Hierarchical Spectral Methodscited by 2
  7. 2023Orthogonal Random Features: Explicit Forms and Sharp Inequalitiescited by 1
  8. 2024One-Bit Quantization and Sparsification for Multiclass Linear Classification with Strong Regularizationcited by 1
  9. 2025A Theory for Conditional Generative Modeling on Multiple Data Sourcescited by 1
  10. 2025On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classificationcited by 1
  11. 2025How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPscited by 1
  12. 2026Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimizationcited by 1
  13. 2026A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computingcited by 1
  14. 2025One-Bit Quantization for Random Features Modelscited by 0
  15. 2025Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networkscited by 0
  16. 2025Universality of high-dimensional scaling limits of stochastic gradient descentcited by 0