mlsanityData to 14 Oct 2026

Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK

arXiv 2301.00327 ↗First posted 1 Jan 2023cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We study training one-hidden-layer ReLU networks in the neural tangent kernel (NTK) regime, where the networks' biases are initialized to some constant rather than zero. We prove that under such initialization, the neural network will have sparse activation throughout the entire training process, which enables fast training procedures via some sophisticated computational methods. With such initialization, we show that the neural networks possess a different limiting kernel which we call textitbias-generalized NTK, and we study various properties of the neural networks with this new kernel. We first characterize the gradient descent dynamics. In particular, we show that the network in this case can achieve as fast convergence as the dense network, as opposed to the previous work suggesting that the sparse networks converge slower. In addition, our result improves the previous required width to ensure convergence. Secondly, we study the networks' generalization: we show a width-sparsity dependence, which yields a sparsity-dependent Rademacher complexity and generalization bound. To our knowledge, this is the first sparsity-dependent generalization result via Rademacher complexity. Lastly, we study the smallest eigenvalue of this new kernel. We identify a data-dependent region where we can derive a much sharper lower bound on the NTK's smallest eigenvalue than the worst-case bound previously known. This can lead to improvement in the generalization bound.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 39 of its 45 references that are in the map. First posted on arXiv 1 Jan 2023.

Builds on

39 of 45 references.

References · 39

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  8. 2018On Lazy Training in Differentiable Programmingcited by 675
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  10. 2017Deep Neural Networks as Gaussian Processescited by 622
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  12. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
27 more
  1. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  2. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  3. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  4. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  5. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  6. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  7. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  8. 2019Linearized two-layers neural networks in high dimensioncited by 155
  9. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
  10. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  11. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  12. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  13. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  14. 2021Sparse is Enough in Scaling Transformerscited by 86
  15. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  16. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  17. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  18. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  19. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 59
  20. 2020Neural Kernels Without Tangentscited by 56
  21. 2021Does Preprocessing Help Training Over-parameterized Neural Networks?cited by 32
  22. 2022Bypass Exponential Time Preprocessing: Fast Neural Network Training via Weight-Data Correlation Preprocessingcited by 29
  23. 2020Finding trainable sparse networks through Neural Tangent Transfercited by 28
  24. 2022Training Overparametrized Neural Networks in Sublinear Timecited by 22
  25. 2022A Sublinear Adversarial Training Algorithmcited by 22
  26. 2022Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysiscited by 15
  27. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Low-Latency Activation-Regularized Sparse Neural Operators with Distillation Assistance Towards Real-Time Neuromorphic Virtual Sensingcited by 1
  2. 2026Why Inference in Large Models Becomes Decomposable After Trainingcited by 0