mlsanityData to 7 Oct 2026

When Are Bias-Free ReLU Networks Effectively Linear Networks?

arXiv 2406.12615 ↗First posted 18 Jun 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We investigate the implications of removing bias in ReLU networks regarding their expressivity and learning dynamics. We first show that two-layer bias-free ReLU networks have limited expressivity: the only odd function two-layer bias-free ReLU networks can express is a linear one. We then show that, under symmetry conditions on the data, these networks have the same learning dynamics as linear networks. This enables us to give analytical time-course solutions to certain two-layer bias-free (leaky) ReLU networks outside the lazy learning regime. While deep bias-free ReLU networks are more expressive than their two-layer counterparts, they still share a number of similarities with deep linear networks. These similarities enable us to leverage insights from linear networks to understand certain ReLU networks. Overall, our results show that some properties previously established for bias-free ReLU networks arise due to equivalence to linear networks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 61 of its 74 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

61 of 74 references.

References · 61

  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2019Benign Overfitting in Linear Regressioncited by 544
  7. 2020Plug-and-Play Image Restoration with Deep Denoiser Priorcited by 467
  8. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  11. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  12. 2018Gradient descent aligns the layers of deep linear networkscited by 182
38 more
  1. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  2. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representationscited by 179
  3. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  4. 2019Linearized two-layers neural networks in high dimensioncited by 155
  5. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  6. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  7. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  8. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  9. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  10. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  11. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  12. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  13. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  14. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  15. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  16. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  17. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  18. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  19. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  20. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  21. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  22. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  23. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  24. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  25. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  26. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  27. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  28. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  29. 2020Separation and Concentration in Deep Networkscited by 28
  30. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  31. 2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
  32. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  33. 2022Exact Solutions of a Deep Linear Networkcited by 24
  34. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  35. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  36. 2018Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalizationcited by 21
  37. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  38. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  2. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  3. 2026The Conflict Between Logic and Memory: Training Conditions for Optimizer-Dependent Rule Acquisitioncited by 0