mlsanityData to 14 Oct 2026

On Generalization Bounds for Neural Networks with Low Rank Layers

arXiv 2411.13733 ↗First posted 20 Nov 2024cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

While previous optimization results have suggested that deep neural networks tend to favour low-rank weight matrices, the implications of this inductive bias on generalization bounds remain underexplored. In this paper, we apply Maurer's chain rule for Gaussian complexity to analyze how low-rank layers in deep networks can prevent the accumulation of rank and dimensionality factors that typically multiply across layers. This approach yields generalization bounds for rank and spectral norm constrained networks. We compare our results to prior generalization bounds for deep networks, highlighting how deep networks with low-rank layers can achieve better generalization than those with full-rank layers. Additionally, we discuss how this framework provides new perspectives on the generalization capabilities of deep networks exhibiting neural collapse.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 20 Nov 2024.

Builds on

20 of 29 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  8. 2019Benign Overfitting in Linear Regressioncited by 549
  9. 2017Implicit Regularization in Matrix Factorizationcited by 322
  10. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  11. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  12. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
8 more
  1. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  2. 2020Directional convergence and alignment in deep learningcited by 178
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  4. 2016A vector-contraction inequality for Rademacher complexitiescited by 142
  5. 2019Harmless interpolation of noisy data in regressioncited by 121
  6. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  7. 2014A chain rule for the expected suprema of Gaussian processescited by 13
  8. 2023A Chain Rule for the Expected Suprema of Bernoulli Processescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
  2. 2026Sharper Generalization Bounds for Transformercited by 3
  3. 2026Bound to Disagree: Generalization Bounds via Certifiable Surrogatescited by 2
  4. 2026Deep Neural Regression Collapsecited by 1