# From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks

arXiv: https://arxiv.org/abs/2602.17596 · HTML: https://mlsanity.com/paper/2602.17596/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Builds on (most cited first, 17 of 17)

- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Topology and Geometry of Half-Rectified Network Optimization](https://mlsanity.com/paper/1611.01540/) (2016)
- [Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets](https://mlsanity.com/paper/1906.06247/) (2019)
- [On Connected Sublevel Sets in Deep Learning](https://mlsanity.com/paper/1901.07417/) (2019)
- [Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks](https://mlsanity.com/paper/1912.10095/) (2019)
- [Proving Linear Mode Connectivity of Neural Networks via Optimal Transport](https://mlsanity.com/paper/2310.19103/) (2023)
- [Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks](https://mlsanity.com/paper/2307.15285/) (2023)
- [Optimal Sets and Solution Paths of ReLU Networks](https://mlsanity.com/paper/2306.00119/) (2023)
- [Understanding Mode Connectivity via Parameter Space Symmetry](https://mlsanity.com/paper/2505.23681/) (2025)
- [Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding](https://mlsanity.com/paper/2402.05626/) (2024)
- [Topological obstruction to the training of shallow ReLU neural networks](https://mlsanity.com/paper/2410.14837/) (2024)
- [Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities](https://mlsanity.com/paper/2602.00693/) (2026)

## Abstract (arXiv)

We study pathwise connectivity of sublevel sets for one-hidden-layer ReLU networks with constrained first-layer weights and an ℓ1 penalty on the output layer. The data term is assumed convex and globally Lipschitz in the scalar logit. We first give a finite-width construction that connects any two points of a common sublevel through a path controlled by a loss-consistent compression functional and a first-order perturbation term. The proof replaces the quadratic perturbation estimate in the Freeman--Bruna mechanism by a direct Lipschitz bound. Positive homogeneity is then used in a direction that is compatible with the penalty: every active atom is moved monotonically from the unit ball to the unit sphere while its output coefficient is reduced. Sphere covering and cluster merging consequently give O(m-1/(n-1)) fixed-level thickening for n≥2, while the one-dimensional two-ray dictionary gives exact connectivity for every m≥4. We also prove internally that the regularized approximation values satisfy e(l)-e_∞=O(l-1/2). More generally, a rate O(l-s) transfers to a near-optimal barrier rate O(m-s/((n-1)s+1)); under the standing assumptions, this yields the explicit rate O(m-1/(n+1)). A theorem-aligned finite-distribution experiment complements the analysis. The primary Huber run yields a maximal best certified upper gap 1.66×10-5 over 720 recorded pairs at widths m≥16; a matched binary-cross-entropy rerun and a 720-endpoint dense-representation stress test probe loss robustness and the active cluster-merging mechanism.