mlsanityData to 7 Oct 2026

Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimation

arXiv 2302.08923 ↗First posted 17 Feb 2023math.ST cond-mat.dis-nn cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

In this manuscript we consider the problem of generalized linear estimation on Gaussian mixture data with labels given by a single-index model. Our first result is a sharp asymptotic expression for the test and training errors in the high-dimensional regime. Motivated by the recent stream of results on the Gaussian universality of the test and training errors in generalized linear estimation, we ask ourselves the question: "when is a single Gaussian enough to characterize the error?". Our formula allow us to give sharp answers to this question, both in the positive and negative directions. More precisely, we show that the sufficient conditions for Gaussian universality (or lack of thereof) crucially depend on the alignment between the target weights and the means and covariances of the mixture clusters, which we precisely quantify. In the particular case of least-squares interpolation, we prove a strong universality property of the training error, and show it follows a simple, closed-form expression. Finally, we apply our results to real datasets, clarifying some recent discussion in the literature about Gaussian universality of the errors in this context.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

31 of 53 references.

References · 31

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  2. 2019Benign Overfitting in Linear Regressioncited by 544
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  5. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  6. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  7. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  8. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  9. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  10. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  11. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  12. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
19 more
  1. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  2. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  3. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  4. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  5. 2022Universality of empirical risk minimizationcited by 73
  6. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  7. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  8. 2018Kymatio: Scattering Transforms in Pythoncited by 54
  9. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  10. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  11. 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 45
  12. 2017The Unreasonable Effectiveness of Structured Random Orthogonal Embeddingscited by 43
  13. 2020The Lasso with general Gaussian designs with applications to hypothesis testingcited by 40
  14. 2022The Implicit Bias of Benign Overfittingcited by 38
  15. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  16. 2020Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic Viewcited by 32
  17. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  18. 2022Learning curves for the multi-class teacher-student perceptroncited by 17
  19. 2022Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional datacited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 13; 2025: 3; 2026: 4.

25 papers.

Sort:

By area:

  1. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  2. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  3. 2024Optimal Ridge Regularization for Out-of-Distribution Predictioncited by 18
  4. 2023High-dimensional robust regression under heavy-tailed data: Asymptotics and Universalitycited by 15
  5. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  6. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  7. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  8. 2024High-dimensional learning of narrow neural networkscited by 12
  9. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  10. 2023The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasetscited by 9
  11. 2024Restoring balance: principled under/oversampling of data for optimal classificationcited by 6
  12. 2025Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Modelscited by 6
13 more
  1. 2024A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offscited by 5
  2. 2024A replica analysis of under-baggingcited by 4
  3. 2024The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixturescited by 4
  4. 2023Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensionscited by 3
  5. 2023Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Modelscited by 3
  6. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  7. 2024On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Boundscited by 2
  8. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 1
  9. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 1
  10. 2026Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimizationcited by 1
  11. 2025Gaussian Universality for Diffusion Modelscited by 0
  12. 2026Precise Performance of Linear Denoisers in the Proportional Regimecited by 0
  13. 2026The sharp SAT/UNSAT phase transition in random ellipsoid fittingcited by 0