# Geometry of Linear Convolutional Networks

arXiv: https://arxiv.org/abs/2108.01538 · HTML: https://mlsanity.com/paper/2108.01538/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 3 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview](https://mlsanity.com/paper/1809.09573/) (2018)
- [Identity Matters in Deep Learning](https://mlsanity.com/paper/1611.04231/) (2016)
- [Deep linear neural networks with arbitrary loss: All local minima are global](https://mlsanity.com/paper/1712.01473/) (2017)
- [Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers](https://mlsanity.com/paper/1910.05505/) (2019)
- [Gradient Dynamics of Shallow Univariate ReLU Networks](https://mlsanity.com/paper/1906.07842/) (2019)
- [Pure and Spurious Critical Points: a Geometric Study of Linear Networks](https://mlsanity.com/paper/1910.01671/) (2019)
- [Depth Creates No Bad Local Minima](https://mlsanity.com/paper/1702.08580/) (2017)
- [The loss surface of deep linear networks viewed through the algebraic geometry lens](https://mlsanity.com/paper/1810.07716/) (2018)
- [The Euclidean distance degree of an algebraic variety](https://mlsanity.com/paper/1309.0049/) (2013)
- [Optimization Theory for ReLU Neural Networks Trained with Normalization Layers](https://mlsanity.com/paper/2006.06878/) (2020)
- [Depth creates no more spurious local minima](https://mlsanity.com/paper/1901.09827/) (2019)
- [Mixtures and products in two graphical models](https://mlsanity.com/paper/1709.05276/) (2017)
- [Wasserstein Distance to Independence Models](https://mlsanity.com/paper/2003.06725/) (2020)
- [Optimal Transport to a Variety](https://mlsanity.com/paper/1909.11716/) (2019)
- [Duality of Multiple Root Loci](https://mlsanity.com/paper/1508.00202/) (2015)

## Built on by (most cited first, 27 of 27)

- [Training Dynamics of In-Context Learning in Linear Attention](https://mlsanity.com/paper/2501.16265/) (2025)
- [Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks](https://mlsanity.com/paper/2201.11729/) (2022)
- [Functional dimension of feedforward ReLU neural networks](https://mlsanity.com/paper/2209.04036/) (2022)
- [Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometry](https://mlsanity.com/paper/2501.18915/) (2025)
- [Function Space and Critical Points of Linear Convolutional Networks](https://mlsanity.com/paper/2304.05752/) (2023)
- [Geometry of Lightning Self-Attention: Identifiability and Dimension](https://mlsanity.com/paper/2408.17221/) (2024)
- [On the Geometry and Optimization of Polynomial Convolutional Networks](https://mlsanity.com/paper/2410.00722/) (2024)
- [Algebraic Complexity and Neurovariety of Linear Convolutional Networks](https://mlsanity.com/paper/2401.16613/) (2024)
- [Geometry and Optimization of Shallow Polynomial Networks](https://mlsanity.com/paper/2501.06074/) (2025)
- [Understanding Mode Connectivity via Parameter Space Symmetry](https://mlsanity.com/paper/2505.23681/) (2025)
- [Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11846/) (2025)
- [An entropy formula for the Deep Linear Network](https://mlsanity.com/paper/2509.09088/) (2025)
- [Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss](https://mlsanity.com/paper/2303.03027/) (2023)
- [Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Space](https://mlsanity.com/paper/2203.00614/) (2022)
- [Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups](https://mlsanity.com/paper/2309.13736/) (2023)
- [Understanding Learning Invariance in Deep Linear Networks](https://mlsanity.com/paper/2506.13714/) (2025)
- [Constraining the outputs of ReLU neural networks](https://mlsanity.com/paper/2508.03867/) (2025)
- [Robustness Verification of Polynomial Neural Networks](https://mlsanity.com/paper/2602.06105/) (2026)
- [The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networks](https://mlsanity.com/paper/2507.06367/) (2025)
- [Causal Representation Meets Stochastic Modeling under Generic Geometry](https://mlsanity.com/paper/2602.05033/) (2026)
- [The Geometry of Polynomial Group Convolutional Neural Networks](https://mlsanity.com/paper/2603.29566/) (2026)
- [Algebraic Representations for Faster Predictions in Convolutional Neural Networks](https://mlsanity.com/paper/2408.07815/) (2024)
- [Algebraic Invariants of Lightning Self-Attention](https://mlsanity.com/paper/2604.15632/) (2026)
- [Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defects](https://mlsanity.com/paper/2605.09609/) (2026)
- [Convergence of gradient flow for learning convolutional neural networks](https://mlsanity.com/paper/2601.08547/) (2026)
- [Singular Learning and Occam's Razor in Deep Monomial Networks](https://mlsanity.com/paper/2606.28464/) (2026)
- [Grassmann--Pl"ucker Parametrization of Convolutional Filter Subspaces: Regularity and Closed Embeddings](https://mlsanity.com/paper/2609.03361/) (2026)

## Abstract (arXiv)

We study the family of functions that are represented by a linear convolutional neural network (LCN). These functions form a semi-algebraic subset of the set of linear maps from input space to output space. In contrast, the families of functions represented by fully-connected linear networks form algebraic sets. We observe that the functions represented by LCNs can be identified with polynomials that admit certain factorizations, and we use this perspective to describe the impact of the network's architecture on the geometry of the resulting function space. We further study the optimization of an objective function over an LCN, analyzing critical points in function space and in parameter space, and describing dynamical invariants for gradient descent. Overall, our theory predicts that the optimized parameters of an LCN will often correspond to repeated filters across layers, or filters that can be decomposed as repeated filters. We also conduct numerical and symbolic experiments that illustrate our results and present an in-depth analysis of the landscape for small architectures.