mlsanityData to 14 Oct 2026

The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program

arXiv 2110.06488 ↗First posted 13 Oct 2021cs.LG math.OCcited by 11

Not on the map. Why?

Abstract (arXiv)

We study non-convex subgradient flows for training two-layer ReLU neural networks from a convex geometry and duality perspective. We characterize the implicit bias of unregularized non-convex gradient flow as convex regularization of an equivalent convex model. We then show that the limit points of non-convex subgradient flows can be identified via primal-dual correspondence in this convex optimization problem. Moreover, we derive a sufficient condition on the dual variables which ensures that the stationary points of the non-convex objective are the KKT points of the convex objective, thus proving convergence of non-convex gradient flows to the global optimum. For a class of regular training data distributions such as orthogonal separable data, we show that this sufficient condition holds. Therefore, non-convex gradient flows in fact converge to optimal solutions of a convex optimization problem. We present numerical results verifying the predictions of our theory for non-convex subgradient descent.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

19 of 25 references.

References · 19

  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  6. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  7. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  8. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  9. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  10. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  11. 2020Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithmscited by 34
  12. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
7 more
  1. 2018Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalizationcited by 21
  2. 2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
  3. 2020Convex Regularization Behind Neural Reconstructioncited by 15
  4. 2021Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Timecited by 14
  5. 2021Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutionscited by 14
  6. 2020The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Modelscited by 8
  7. 2021Training Quantized Neural Networks to Global Optimality via Semidefinite Programmingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 2; 2025: 1; 2026: 1.

11 papers.

  1. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  2. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  3. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  4. 2022Adversarial Reprogramming Revisitedcited by 10
  5. 2024Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimizationcited by 7
  6. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6
  7. 2023From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexitycited by 4
  8. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 4
  9. 2022Overparameterized ReLU Neural Networks Learn the Simplest Models: Neural Isometry and Exact Recoverycited by 3
  10. 2023Convex Dual Theory Analysis of Two-Layer Convolutional Neural Networks with Soft-Thresholdingcited by 0
  11. 2026Convex Optimization of Spiking Neural Networks with Global and Approximation Guaranteescited by 0