mlsanityData to 14 Oct 2026

Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis

arXiv 2001.03371 ↗First posted 10 Jan 2020stat.ML cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

The plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. The phenomenon is actively inspected in the 1990s and found to be due to the fundamental hierarchical structure of neural network models. Then the phenomenon has been thought as inevitable. However, the phenomenon seldom occurs in the context of recent deep learning. There is a gap between theory and reality. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned. It is shown that the data whose covariance has small and dispersed eigenvalues tend to make the plateau phenomenon inconspicuous.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 2; 2024: 4; 2025: 7; 2026: 2.

23 papers.

Sort:

By area:

  1. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  2. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  3. 2023Learning time-scales in two-layers neural networkscited by 57
  4. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  5. 2022Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activationcited by 16
  6. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  7. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  8. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  9. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  10. 2021Disentangled Variational Information Bottleneck for Multiview Representation Learningcited by 8
  11. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
  12. 2022Learning and generalization of one-hidden-layer neural networks, going beyond standard Gaussian datacited by 8
11 more
  1. 2021Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machinescited by 7
  2. 2024How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalancecited by 6
  3. 2025Reheated Gradient-based Discrete Sampling for Combinatorial Optimizationcited by 6
  4. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  5. 2025Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learningcited by 5
  6. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
  7. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  8. 2025Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokkingcited by 2
  9. 2023Online Learning for the Random Feature Model in the Student-Teacher Frameworkcited by 0
  10. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  11. 2026Statistical Vignettes on the Grokking Phenomenoncited by 0