# On the Convergence of Overparameterized Problems: Inherent Properties of the Compositional Structure of Neural Networks

arXiv: https://arxiv.org/abs/2511.09810 · HTML: https://mlsanity.com/paper/2511.09810/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 10 of 10)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers](https://mlsanity.com/paper/1910.05505/) (2019)
- [Training Linear Neural Networks: Non-Local Convergence and Complexity Results](https://mlsanity.com/paper/2002.09852/) (2020)
- [The geometry of the deep linear network](https://mlsanity.com/paper/2411.09004/) (2024)
- [A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks](https://mlsanity.com/paper/1811.03568/) (2018)
- [Convergence Analysis of Gradient Flow for Overparameterized LQR Formulations](https://mlsanity.com/paper/2408.15456/) (2024)

## Abstract (arXiv)

This paper investigates how the compositional structure of neural networks shapes their optimization landscape and training dynamics. We analyze the gradient flow associated with overparameterized optimization problems, which can be interpreted as training a neural network with linear activations. Remarkably, we show that the global convergence properties can be derived for any cost function that is proper and real analytic. We then specialize the analysis to scalar-valued cost functions, where the geometry of the landscape can be fully characterized. In this setting, we demonstrate that key structural features -- such as the location and stability of saddle points -- are universal across all admissible costs, depending solely on the overparameterized representation rather than on problem-specific details. Moreover, we show that convergence can be arbitrarily accelerated depending on the initialization, as measured by an imbalance metric introduced in this work. Finally, we discuss how these insights may generalize to neural networks with sigmoidal activations, showing through a simple example which geometric and dynamical properties persist beyond the linear case.