# Implicit Bias of Linear Equivariant Networks

arXiv: https://arxiv.org/abs/2110.06084 · HTML: https://mlsanity.com/paper/2110.06084/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 25 of 25)

- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018)
- [Cormorant: Covariant Molecular Neural Networks](https://mlsanity.com/paper/1906.04015/) (2019)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network](https://mlsanity.com/paper/1806.09231/) (2018)
- [Universal approximations of invariant maps by neural networks](https://mlsanity.com/paper/1804.10306/) (2018)
- [Implicit Regularization in Deep Learning May Not Be Explainable by Norms](https://mlsanity.com/paper/2005.06398/) (2020)
- [Provably Strict Generalisation Benefit for Equivariant Models](https://mlsanity.com/paper/2102.10333/) (2021)
- [On the Benefits of Invariance in Neural Networks](https://mlsanity.com/paper/2005.00178/) (2020)
- [A Unifying View on Implicit Bias in Training Linear Neural Networks](https://mlsanity.com/paper/2010.02501/) (2020)
- [Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm](https://mlsanity.com/paper/2102.12238/) (2021)
- [Generalization capabilities of translationally equivariant neural networks](https://mlsanity.com/paper/2103.14686/) (2021)
- [Local Convolutions Cause an Implicit Bias towards High Frequency Adversarial Examples](https://mlsanity.com/paper/2006.11440/) (2020)
- [Generalized moduli of continuity under irregular or random deformations via multiscale analysis](https://mlsanity.com/paper/2104.11977/) (2021)

## Built on by (most cited first, 19 of 19)

- [The Exact Sample Complexity Gain from Invariances for Kernel Regression](https://mlsanity.com/paper/2303.14269/) (2023)
- [projUNN: efficient method for training deep networks with unitary matrices](https://mlsanity.com/paper/2203.05483/) (2022)
- [Optimization Dynamics of Equivariant and Augmented Neural Networks](https://mlsanity.com/paper/2303.13458/) (2023)
- [A Universal Class of Sharpness-Aware Minimization Algorithms](https://mlsanity.com/paper/2406.03682/) (2024)
- [Machine learning and invariant theory](https://mlsanity.com/paper/2209.14991/) (2022)
- [On the hardness of learning under symmetries](https://mlsanity.com/paper/2401.01869/) (2024)
- [Towards fully covariant machine learning](https://mlsanity.com/paper/2301.13724/) (2023)
- [On the Implicit Bias of Linear Equivariant Steerable Networks](https://mlsanity.com/paper/2303.04198/) (2023)
- [Data Augmentation and Regularization for Learning Group Equivariance](https://mlsanity.com/paper/2502.06547/) (2025)
- [Spectral Graph Neural Networks are Incomplete on Graphs with a Simple Spectrum](https://mlsanity.com/paper/2506.05530/) (2025)
- [Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics](https://mlsanity.com/paper/2512.04006/) (2025)
- [A tradeoff between universality of equivariant models and learnability of symmetries](https://mlsanity.com/paper/2210.09444/) (2022)
- [Testing predictions of representation cost theory with CNNs](https://mlsanity.com/paper/2210.01257/) (2022)
- [Learning Decentralized Swarms Using Rotation Equivariant Graph Neural Networks](https://mlsanity.com/paper/2502.17612/) (2025)
- [Equivariant symmetry-aware head pose estimation for fetal MRI](https://mlsanity.com/paper/2512.04890/) (2025)
- [Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power](https://mlsanity.com/paper/2512.09673/) (2025)
- [The Implicit Bias of Depth: From Neural Collapse to Softmax Codes](https://mlsanity.com/paper/2605.23087/) (2026)
- [Equivariance and Augmentation for Bayesian Neural Networks](https://mlsanity.com/paper/2606.26273/) (2026)
- [Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines](https://mlsanity.com/paper/2610.09196/) (2026)

## Abstract (arXiv)

Group equivariant convolutional neural networks (G-CNNs) are generalizations of convolutional neural networks (CNNs) which excel in a wide range of technical applications by explicitly encoding symmetries, such as rotations and permutations, in their architectures. Although the success of G-CNNs is driven by their explicit symmetry bias, a recent line of work has proposed that the implicit bias of training algorithms on particular architectures is key to understanding generalization for overparameterized neural nets. In this context, we show that L-layer full-width linear G-CNNs trained via gradient descent for binary classification converge to solutions with low-rank Fourier matrix coefficients, regularized by the 2/L-Schatten matrix norm. Our work strictly generalizes previous analysis on the implicit bias of linear CNNs to linear G-CNNs over all finite groups, including the challenging setting of non-commutative groups (such as permutations), as well as band-limited G-CNNs over infinite groups. We validate our theorems via experiments on a variety of groups, and empirically explore more realistic nonlinear networks, which locally capture similar regularization patterns. Finally, we provide intuitive interpretations of our Fourier space implicit regularization results in real space via uncertainty principles.