Data to 7 Oct 2026

How do infinite width bounded norm networks look in function space?

arXiv 1902.05040 ↗First posted 13 Feb 2019cs.LG stat.MLcited by 105

Not on the map. Why?

Abstract (arXiv)

We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of squares of all weights in the system, except for an unregularized bias term for each unit) is bounded; or equivalently what is the minimal norm required to approximate a given function. For functions f : mathbb R rightarrow mathbb R and a single hidden layer, we show that the minimal network norm for representing f is max(int |f''(x)| dx, |f'(-∞) + f'(+∞)|), and hence the minimal norm fit for a sample is given by a linear spline interpolation.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 23; 2023: 22; 2024: 16; 2025: 10; 2026: 14.

105 papers.

Sort:

By area:

2022Learning Single-Index Models with Shallow Neural Networkscited by 92
2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
2021Understanding neural networks with reproducing kernel Banach spacescited by 50
2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 40
2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
2021Scaled ReLU Matters for Training Vision Transformerscited by 29
2022Flat minima generalize for low-rank matrix recoverycited by 28

38 more

2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
2024When does compositional structure yield compositional generalization? A kernel theorycited by 19
2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
2023Penalising the biases in norm regularisation enforces sparsitycited by 17
2023Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compressioncited by 17
2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 15
2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
2025On the Interpolation Effect of Score Smoothing in Diffusion Modelscited by 15
2022Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysiscited by 14
2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
2022Knowledge Distillation: Bad Models Can Be Good Role Modelscited by 12
2022Convexifying Transformers: Improving optimization and understanding of transformer networkscited by 12
2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
2022Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groupscited by 11
2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 11
2021Deep Learning Generalization and the Convex Hull of Training Setscited by 10
2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
2023How do Minimum-Norm Shallow Denoisers Look in Function Space?cited by 9
2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
2021On Energy-Based Models with Overparametrized Shallow Neural Networkscited by 8
2021Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functionscited by 8
2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
2021Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Spacecited by 7
2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 7
2023The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Modelscited by 7
2025Scaling can lead to compositional generalizationcited by 7
2023Extending Path-Dependent NJ-ODEs to Noisy Observations and a Dependent Observation Frameworkcited by 6
2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6

50 most cited of 105 All 105 (JSON)