mlsanityData to 14 Oct 2026

Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization

arXiv 2307.12851 ↗First posted 24 Jul 2023cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

This paper studies the problem of training a two-layer ReLU network for binary classification using gradient flow with small initialization. We consider a training dataset with well-separated input vectors: Any pair of input data with the same label are positively correlated, and any pair with different labels are negatively correlated. Our analysis shows that, during the early phase of training, neurons in the first layer try to align with either the positive data or the negative data, depending on its corresponding weight on the second layer. A careful analysis of the neurons' directional dynamics allows us to provide an O(fraclog n√μ) upper bound on the time it takes for all neurons to achieve good alignment with the input data, where n is the number of data points and μ measures how well the data are separated. After the early alignment phase, the loss converges to zero at a O(frac1t) rate, and the weight matrix on the first layer is approximately low-rank. Numerical experiments on the MNIST dataset illustrate our theoretical findings.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

25 of 44 references.

References · 25

  1. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  2. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  3. 2017Implicit Regularization in Matrix Factorizationcited by 322
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  5. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  6. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  7. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  8. 2020Directional convergence and alignment in deep learningcited by 178
  9. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  10. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  11. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  12. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
13 more
  1. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  2. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  3. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  4. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  5. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  6. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  7. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  8. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  9. 2018Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalizationcited by 21
  10. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 20
  11. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  12. 2021The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Programcited by 11
  13. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 12; 2026: 4.

28 papers.

Sort:

By area:

  1. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  2. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 19
  3. 2025An overview of condensation phenomenon in deep learningcited by 17
  4. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  5. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  6. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  7. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 11
  8. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  9. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  10. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  11. 2024Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializationscited by 7
  12. 2024Can Implicit Bias Imply Adversarial Robustness?cited by 5
16 more
  1. 2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5
  2. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 5
  3. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  4. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 4
  5. 2025Low Rank Gradients and Where to Find Themcited by 4
  6. 2025Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Datacited by 2
  7. 2024On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stagescited by 1
  8. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 1
  9. 2025Understanding Incremental Learning with Closed-form Solution to Gradient Flow on Overparamerterized Matrix Factorizationcited by 1
  10. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  11. 2025Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networkscited by 0
  12. 2025KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networkscited by 0
  13. 2025An Equivariance Toolbox for Learning Dynamicscited by 0
  14. 2026Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalizationcited by 0
  15. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  16. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0