mlsanityData to 7 Oct 2026

Function Space and Critical Points of Linear Convolutional Networks

arXiv 2304.05752 ↗First posted 12 Apr 2023cs.LG math.AGcited by 19

Not on the map. Why?

Abstract (arXiv)

We study the geometry of linear networks with one-dimensional convolutional layers. The function spaces of these networks can be identified with semi-algebraic families of polynomials admitting sparse factorizations. We analyze the impact of the network's architecture on the function space's dimension, boundary, and singular points. We also describe the critical points of the network's parameterization map. Furthermore, we study the optimization problem of training a network with the squared error loss. We prove that for architectures where all strides are larger than one and generic data, the non-zero critical points of that optimization problem are smooth interior points of the function space. This property is known to be false for dense linear networks and linear convolutional networks with stride one.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 15 of its 37 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

15 of 37 references.

References · 15

  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  2. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  3. 2016Deep Learning without Poor Local Minimacited by 233
  4. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 96
  5. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 65
  6. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  7. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  8. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  9. 2017Depth Creates No Bad Local Minimacited by 29
  10. 2021Geometry of Linear Convolutional Networkscited by 27
  11. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  12. 2018The loss surface of deep linear networks viewed through the algebraic geometry lenscited by 20
3 more
  1. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  2. 2018The Global Optimization Geometry of Shallow Linear Neural Networkscited by 11
  3. 2023Complex Critical Points of Deep Linear Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 8; 2026: 6.

19 papers.

Sort:

  1. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  2. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  3. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  4. 2024Activation degree thresholds and expressiveness of polynomial neural networkscited by 14
  5. 2024On the Geometry and Optimization of Polynomial Convolutional Networkscited by 11
  6. 2024Algebraic Complexity and Neurovariety of Linear Convolutional Networkscited by 10
  7. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  8. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  9. 2025An entropy formula for the Deep Linear Networkcited by 8
  10. 2023Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groupscited by 5
  11. 2025Understanding Learning Invariance in Deep Linear Networkscited by 3
  12. 2025Constraining the outputs of ReLU neural networkscited by 3
7 more
  1. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  2. 2026Causal Representation Meets Stochastic Modeling under Generic Geometrycited by 2
  3. 2026The Geometry of Polynomial Group Convolutional Neural Networkscited by 2
  4. 2026Algebraic Invariants of Lightning Self-Attentioncited by 1
  5. 2026Convergence of gradient flow for learning convolutional neural networkscited by 0
  6. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  7. 2026Grassmann--Pl"ucker Parametrization of Convolutional Filter Subspaces: Regularity and Closed Embeddingscited by 0