# Representation and Regression Problems in Neural Networks: Relaxation, Generalization, and Numerics

arXiv: https://arxiv.org/abs/2412.01619 · HTML: https://mlsanity.com/paper/2412.01619/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 62 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 23 of 23)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [The Power of Depth for Feedforward Neural Networks](https://mlsanity.com/paper/1512.03965/) (2015)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity](https://mlsanity.com/paper/1810.07770/) (2018)
- [A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks](https://mlsanity.com/paper/2001.11443/) (2020)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [A representer theorem for deep neural networks](https://mlsanity.com/paper/1802.09210/) (2018)
- [Sparsity of solutions for variational inverse problems with finite-dimensional data](https://mlsanity.com/paper/1809.05045/) (2018)
- [On Representer Theorems and Convex Regularization](https://mlsanity.com/paper/1806.09810/) (2018)
- [Exact Support Recovery for Sparse Spikes Deconvolution](https://mlsanity.com/paper/1306.6909/) (2013)
- [On Sparsity in Overparametrised Shallow ReLU Networks](https://mlsanity.com/paper/2006.10225/) (2020)
- [Control of neural transport for normalizing flows](https://mlsanity.com/paper/2307.07817/) (2023)
- [Interplay between depth and width for interpolation in neural ODEs](https://mlsanity.com/paper/2401.09902/) (2024)
- [When Expressivity Meets Trainability: Fewer than n Neurons Can Work](https://mlsanity.com/paper/2210.12001/) (2022)
- [Constructive Universal Approximation and Finite Sample Memorization by Narrow Deep ReLU Networks](https://mlsanity.com/paper/2409.06555/) (2024)
- [Large-scale nonconvex optimization: randomization, gap estimation, and numerical resolution](https://mlsanity.com/paper/2204.02366/) (2022)

## Built on by (most cited first, 2 of 2)

- [Machine Learning and Control: Foundations, Advances, and Perspectives](https://mlsanity.com/paper/2510.03303/) (2025)
- [Long-Time Trajectory Approximation via SA-NODEs: Model Predictive and Floquet Strategies](https://mlsanity.com/paper/2608.10738/) (2026)

## Abstract (arXiv)

In this work, we address three non-convex optimization problems associated with the training of shallow neural networks (NNs) for exact and approximate representation, as well as for regression tasks. Through a mean-field approach, we convexify these problems and, applying a representer theorem, prove the absence of relaxation gaps. We establish generalization bounds for the resulting NN solutions, assessing their predictive performance on test datasets and, analyzing the impact of key hyperparameters on these bounds, propose optimal choices. On the computational side, we examine the discretization of the convexified problems and derive convergence rates. For low-dimensional datasets, these discretized problems are efficiently solvable using the simplex method. For high-dimensional datasets, we propose a sparsification algorithm that, combined with gradient descent for over-parameterized shallow NNs, yields effective solutions to the primal problems.