# A Dual Certificate Approach to Sparsity in Infinite-Width Shallow Neural Networks

arXiv: https://arxiv.org/abs/2603.17785 · HTML: https://mlsanity.com/paper/2603.17785/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Builds on (most cited first, 25 of 25)

- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case](https://mlsanity.com/paper/1910.01635/) (2019)
- [How do infinite width bounded norm networks look in function space?](https://mlsanity.com/paper/1902.05040/) (2019)
- [Representation formulas and pointwise properties for Barron functions](https://mlsanity.com/paper/2006.05982/) (2020)
- [Gradient Descent Quantizes ReLU Network Features](https://mlsanity.com/paper/1803.08367/) (2018)
- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [Understanding neural networks with reproducing kernel Banach spaces](https://mlsanity.com/paper/2109.09710/) (2021)
- [Sparsity of solutions for variational inverse problems with finite-dimensional data](https://mlsanity.com/paper/1809.05045/) (2018)
- [On Representer Theorems and Convex Regularization](https://mlsanity.com/paper/1806.09810/) (2018)
- [Exact Support Recovery for Sparse Spikes Deconvolution](https://mlsanity.com/paper/1306.6909/) (2013)
- [Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape](https://mlsanity.com/paper/2305.19510/) (2023)
- [The Sliding Frank-Wolfe Algorithm and its Application to Super-Resolution Microscopy](https://mlsanity.com/paper/1811.06416/) (2018)
- [Some observations on high-dimensional partial differential equations with Barron data](https://mlsanity.com/paper/2012.01484/) (2020)
- [Duality for Neural Networks through Reproducing Kernel Banach Spaces](https://mlsanity.com/paper/2211.05020/) (2022)
- [Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks](https://mlsanity.com/paper/2111.02278/) (2021)
- [Exploring the loss landscape of regularized neural networks via convex duality](https://mlsanity.com/paper/2411.07729/) (2024)
- [On Sparsity in Overparametrised Shallow ReLU Networks](https://mlsanity.com/paper/2006.10225/) (2020)
- [Superresolution without Separation](https://mlsanity.com/paper/1506.03144/) (2015)
- [Embeddings between Barron spaces with higher order activation functions](https://mlsanity.com/paper/2305.15839/) (2023)
- [A Lipschitz spaces view of infinitely wide shallow neural networks](https://mlsanity.com/paper/2410.14591/) (2024)
- [Vector-Valued Reproducing Kernel Banach Spaces for Neural Networks and Operators](https://mlsanity.com/paper/2509.26371/) (2025)
- [A sparse control approach to optimal sensor placement in PDE-constrained parameter estimation problems](https://mlsanity.com/paper/1905.01696/) (2019)
- [A General Theory for Exact Sparse Representation Recovery in Convex Optimization](https://mlsanity.com/paper/2311.08072/) (2023)
- [Sampling Theory for Super-Resolution with Implicit Neural Representations](https://mlsanity.com/paper/2506.09949/) (2025)

## Built on by (most cited first, 1 of 1)

- [Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse](https://mlsanity.com/paper/2607.03613/) (2026)

## Abstract (arXiv)

In this paper, we study total variation (TV)-regularized training of infinite-width shallow ReLU neural networks, formulated as a convex optimization problem over measures on the unit sphere. Our approach leverages the duality theory of TV-regularized optimization problems to establish rigorous guarantees on the sparsity of the solutions to the training problem. Our analysis further characterizes how and when this sparsity persists in a low noise regime and for small regularization parameter. The key observation that motivates our analysis is that, for ReLU activations, the associated dual certificate is piecewise linear in the weight space. Its linearity regions, which we name dual regions, are determined by the activation patterns of the data via the induced hyperplane arrangement. Taking advantage of this structure, we prove that, on each dual region, the dual certificate admits at most one extreme value. As a consequence, the support of any minimizer is finite, and its cardinality can be bounded from above by a constant depending only on the geometry of the data-induced hyperplane arrangement. Then, we further investigate sufficient conditions ensuring uniqueness of such sparse solution. Finally, under a suitable non-degeneracy condition on the dual certificate along the boundaries of the dual regions, we prove that in the presence of low label noise and for small regularization parameter, solutions to the training problem remain sparse with the same number of Dirac deltas. Additionally, their location and the amplitudes converge, and, in case the locations lie in the interior of a dual region, the convergence happens with a rate that depends linearly on the noise and the regularization parameter.