mlsanityData to 14 Oct 2026

Gaussian Universality of Perceptrons with Random Labels

arXiv 2205.13303 ↗First posted 26 May 2022stat.ML cond-mat.dis-nn cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

While classical in many theoretical settings - and in particular in statistical physics-inspired works - the assumption of Gaussian i.i.d. input data is often perceived as a strong limitation in the context of statistics and machine learning. In this study, we redeem this line of work in the case of generalized linear classification, a.k.a. the perceptron model, with random labels. We argue that there is a large universality class of high-dimensional input data for which we obtain the same minimum training loss as for Gaussian data with corresponding data covariance. In the limit of vanishing regularization, we further demonstrate that the training loss is independent of the data covariance. On the theoretical side, we prove this universality for an arbitrary mixture of homogeneous Gaussian clouds. Empirically, we show that the universality holds also for a broad range of real datasets.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 45 of its 99 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

45 of 99 references.

References · 45

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  6. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  7. 2017A Closer Look at Memorization in Deep Networkscited by 942
  8. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  9. 2018On Lazy Training in Differentiable Programmingcited by 675
  10. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  11. 2017Exploring Generalization in Deep Learningcited by 600
  12. 2019Benign Overfitting in Linear Regressioncited by 549
33 more
  1. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  2. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  3. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  4. 2017A Rewriting System for Convex Optimization Problemscited by 179
  5. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  6. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  7. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  8. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  9. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  10. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  11. 2017A Random Matrix Approach to Neural Networkscited by 80
  12. 2022Universality of empirical risk minimizationcited by 75
  13. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  14. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  15. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 59
  16. 2018Kymatio: Scattering Transforms in Pythoncited by 58
  17. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  18. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
  19. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  20. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  21. 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 46
  22. 2020Provable tradeoffs in adversarially robust classificationcited by 40
  23. 2020The Lasso with general Gaussian designs with applications to hypothesis testingcited by 40
  24. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  25. 2021Probing transfer learning with a model of synthetic correlated datasetscited by 25
  26. 2021Phase Transitions in Transfer Learning for High-Dimensional Perceptronscited by 22
  27. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
  28. 2019Asymptotic Bayes risk for Gaussian mixture in a semi-supervised settingcited by 15
  29. 2016Phase transitions and optimal algorithms in high-dimensional Gaussian mixture clusteringcited by 13
  30. 2020Algorithmic pure states for the negative spherical perceptroncited by 13
  31. 2021Tractability from overparametrization: The example of the negative perceptroncited by 12
  32. 2022Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional datacited by 12
  33. 2019Rademacher complexity and spin glasses: A link between the replica and statistical theories of learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 11; 2025: 3; 2026: 7.

26 papers.

Sort:

By area:

  1. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  2. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  3. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 16
  4. 2023High-dimensional robust regression under heavy-tailed data: Asymptotics and Universalitycited by 15
  5. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  6. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  7. 2024High-dimensional learning of narrow neural networkscited by 12
  8. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  9. 2023The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasetscited by 9
  10. 2024A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offscited by 7
  11. 2024Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networkscited by 6
  12. 2024The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixturescited by 4
14 more
  1. 2024On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Boundscited by 4
  2. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
  3. 2024Effect of Weight Quantization on Learning Models by Typical Case Analysiscited by 3
  4. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  5. 2025On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classificationcited by 2
  6. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  7. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 2
  8. 2026Lasso Universality Under Linearly Dependent Covariates in the Sparse Regimecited by 2
  9. 2024On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networkscited by 1
  10. 2026Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimizationcited by 1
  11. 2026Factual recall in linear associative memories: sharp asymptotics and mechanistic insightscited by 1
  12. 2026The sharp SAT/UNSAT phase transition in random ellipsoid fittingcited by 1
  13. 2026Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networkscited by 1
  14. 2025Gaussian Universality for Diffusion Modelscited by 0