mlsanityData to 14 Oct 2026

Learning curves for the multi-class teacher-student perceptron

arXiv 2203.12094 ↗First posted 22 Mar 2022stat.ML cond-mat.dis-nn cs.LGcited by 17

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

One of the most classical results in high-dimensional learning theory provides a closed-form expression for the generalisation error of binary classification with the single-layer teacher-student perceptron on i.i.d. Gaussian inputs. Both Bayes-optimal estimation and empirical risk minimisation (ERM) were extensively analysed for this setting. At the same time, a considerable part of modern machine learning practice concerns multi-class classification. Yet, an analogous analysis for the corresponding multi-class teacher-student perceptron was missing. In this manuscript we fill this gap by deriving and evaluating asymptotic expressions for both the Bayes-optimal and ERM generalisation errors in the high-dimensional regime. For Gaussian teacher weights, we investigate the performance of ERM with both cross-entropy and square losses, and explore the role of ridge regularisation in approaching Bayes-optimality. In particular, we observe that regularised cross-entropy minimisation yields close-to-optimal accuracy. Instead, for a binary teacher we show that a first-order phase transition arises in the Bayes-optimal performance.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 15 of its 47 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

15 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Learning curves of generic features maps for realistic datasets with a teacher-student model16 Feb

Other references · 14

  1. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  2. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  3. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
  4. 2021The Shape of Learning Curves: a Reviewcited by 70
  5. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  6. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  7. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  8. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  9. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
  10. 2021Graph-based Approximate Message Passing Iterationscited by 35
  11. 2020The estimation error of general first order methodscited by 33
  12. 2020Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic Viewcited by 32
2 more
  1. 2020Analytic Study of Double Descent in Binary Classification: The Impact of Losscited by 28
  2. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 6; 2025: 0; 2026: 1.

17 papers.

Sort:

By area:

  1. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  2. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 26
  3. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 16
  4. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  5. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  6. 2024High-dimensional learning of narrow neural networkscited by 12
  7. 2022Bias-inducing geometries: an exactly solvable data model with fairness implicationscited by 9
  8. 2023Phase transitions in the mini-batch size for sparse and dense two-layer neural networkscited by 9
  9. 2023Wigner kernels: body-ordered equivariant machine learning without a basiscited by 7
  10. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  11. 2023Neural-prior stochastic block modelcited by 4
  12. 2023Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensionscited by 3
5 more
  1. 2023Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Modelscited by 3
  2. 2026A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classificationcited by 3
  3. 2024Nonlinear classification of neural manifolds with contextual informationcited by 2
  4. 2023The Copycat Perceptron: Smashing Barriers Through Collective Learningcited by 1
  5. 2024A Convergence Analysis of Approximate Message Passing with Non-Separable Functions and Applications to Multi-Class Classificationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024The effect of priors on Learning with Restricted Boltzmann Machines
  2. 2025Generalization performance of narrow one-hidden layer networks in the teacher-student setting
  3. 2024Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens
  4. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
  5. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
  6. 2023High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling
  7. 2025Incorporating priors in learning: a random matrix study under a teacher-student framework
  8. 2022Toward Student-Oriented Teacher Network Training For Knowledge Distillation