mlsanityData to 7 Oct 2026

Geometry of Linear Convolutional Networks

arXiv 2108.01538 ↗First posted 3 Aug 2021cs.LG math.AGcited by 27

Not on the map. Why?

Abstract (arXiv)

We study the family of functions that are represented by a linear convolutional neural network (LCN). These functions form a semi-algebraic subset of the set of linear maps from input space to output space. In contrast, the families of functions represented by fully-connected linear networks form algebraic sets. We observe that the functions represented by LCNs can be identified with polynomials that admit certain factorizations, and we use this perspective to describe the impact of the network's architecture on the geometry of the resulting function space. We further study the optimization of an objective function over an LCN, analyzing critical points in function space and in parameter space, and describing dynamical invariants for gradient descent. Overall, our theory predicts that the optimized parameters of an LCN will often correspond to repeated filters across layers, or filters that can be decomposed as repeated filters. We also conduct numerical and symbolic experiments that illustrate our results and present an in-depth analysis of the landscape for small architectures.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 23 of its 43 references that are in the map. First posted on arXiv 3 Aug 2021.

Builds on

23 of 43 references.

References · 23

  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  6. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  7. 2016Deep Learning without Poor Local Minimacited by 233
  8. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  9. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  10. 2016Identity Matters in Deep Learningcited by 107
  11. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  12. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
11 more
  1. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  2. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  3. 2017Depth Creates No Bad Local Minimacited by 29
  4. 2018The loss surface of deep linear networks viewed through the algebraic geometry lenscited by 20
  5. 2013The Euclidean distance degree of an algebraic varietycited by 15
  6. 2020Optimization Theory for ReLU Neural Networks Trained with Normalization Layerscited by 15
  7. 2019Depth creates no more spurious local minimacited by 5
  8. 2017Mixtures and products in two graphical modelscited by 4
  9. 2020Wasserstein Distance to Independence Modelscited by 4
  10. 2019Optimal Transport to a Varietycited by 3
  11. 2015Duality of Multiple Root Locicited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 4; 2025: 9; 2026: 8.

27 papers.

Sort:

  1. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  2. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  3. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  4. 2022Functional dimension of feedforward ReLU neural networkscited by 20
  5. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  6. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  7. 2024On the Geometry and Optimization of Polynomial Convolutional Networkscited by 11
  8. 2024Algebraic Complexity and Neurovariety of Linear Convolutional Networkscited by 10
  9. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  10. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  11. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  12. 2025An entropy formula for the Deep Linear Networkcited by 8
15 more
  1. 2023Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Losscited by 7
  2. 2022Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Spacecited by 6
  3. 2023Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groupscited by 5
  4. 2025Understanding Learning Invariance in Deep Linear Networkscited by 3
  5. 2025Constraining the outputs of ReLU neural networkscited by 3
  6. 2026Robustness Verification of Polynomial Neural Networkscited by 3
  7. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  8. 2026Causal Representation Meets Stochastic Modeling under Generic Geometrycited by 2
  9. 2026The Geometry of Polynomial Group Convolutional Neural Networkscited by 2
  10. 2024Algebraic Representations for Faster Predictions in Convolutional Neural Networkscited by 1
  11. 2026Algebraic Invariants of Lightning Self-Attentioncited by 1
  12. 2026Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defectscited by 1
  13. 2026Convergence of gradient flow for learning convolutional neural networkscited by 0
  14. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  15. 2026Grassmann--Pl"ucker Parametrization of Convolutional Filter Subspaces: Regularity and Closed Embeddingscited by 0