# The Implicit Bias of Gradient Descent on Generalized Gated Linear Networks

arXiv: https://arxiv.org/abs/2202.02649 · HTML: https://mlsanity.com/paper/2202.02649/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 5 Feb 2022.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [A Rewriting System for Convex Optimization Problems](https://mlsanity.com/paper/1709.04494/) (2017)
- [OSQP: An Operator Splitting Solver for Quadratic Programs](https://mlsanity.com/paper/1711.08013/) (2017)
- [Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models](https://mlsanity.com/paper/1905.07325/) (2019)
- [Gated Linear Networks](https://mlsanity.com/paper/1910.01526/) (2019)
- [Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning](https://mlsanity.com/paper/2006.10529/) (2020)
- [Online Learning with Gated Linear Networks](https://mlsanity.com/paper/1712.01897/) (2017)

## Built on by (most cited first, 3 of 3)

- [Globally Gated Deep Linear Networks](https://mlsanity.com/paper/2210.17449/) (2022)
- [A statistical physics framework for optimal learning](https://mlsanity.com/paper/2507.07907/) (2025)
- [Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignment](https://mlsanity.com/paper/2602.12384/) (2026)

## Abstract (arXiv)

Understanding the asymptotic behavior of gradient-descent training of deep neural networks is essential for revealing inductive biases and improving network performance. We derive the infinite-time training limit of a mathematically tractable class of deep nonlinear neural networks, gated linear networks (GLNs), and generalize these results to gated networks described by general homogeneous polynomials. We study the implications of our results, focusing first on two-layer GLNs. We then apply our theoretical predictions to GLNs trained on MNIST and show how architectural constraints and the implicit bias of gradient descent affect performance. Finally, we show that our theory captures a substantial portion of the inductive bias of ReLU networks. By making the inductive bias explicit, our framework is poised to inform the development of more efficient, biologically plausible, and robust learning algorithms.