mlsanityData to 7 Oct 2026

Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines

arXiv 2104.14546 ↗First posted 29 Apr 2021cond-mat.dis-nn cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Over-parametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of over-parametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For on-line learning of a two-layer soft committee machine in the over-realizable case, we find that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 10 of its 35 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

10 of 35 references.

References · 10

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  8. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  9. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 22
  10. 2019Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 2; 2026: 1.

7 papers.

  1. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  2. 2023How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initializationcited by 18
  3. 2024Toward Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixture Modelscited by 6
  4. 2026A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classificationcited by 3
  5. 2025Analytic theory of dropout regularizationcited by 2
  6. 2025Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussiancited by 2
  7. 2023Online Learning for the Random Feature Model in the Student-Teacher Frameworkcited by 0