# Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss

arXiv: https://arxiv.org/abs/2402.00152 · HTML: https://mlsanity.com/paper/2402.00152/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 55 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Builds on (most cited first, 30 of 31)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Towards Deep Neural Network Architectures Robust to Adversarial Examples](https://mlsanity.com/paper/1412.5068/) (2014)
- [Sobolev Training for Neural Networks](https://mlsanity.com/paper/1706.04859/) (2017)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [The phase diagram of approximation rates for deep neural networks](https://mlsanity.com/paper/1906.09477/) (2019)
- [Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networks](https://mlsanity.com/paper/2101.12365/) (2021)
- [Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs](https://mlsanity.com/paper/2106.14473/) (2021)
- [Banach Wasserstein GAN](https://mlsanity.com/paper/1806.06621/) (2018)
- [Generalization Error in Deep Learning](https://mlsanity.com/paper/1808.01174/) (2018)
- [Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations](https://mlsanity.com/paper/1809.03062/) (2018)
- [Nonlinear Approximation via Compositions](https://mlsanity.com/paper/1902.10170/) (2019)
- [Deep Model Compression: Distilling Knowledge from Noisy Teachers](https://mlsanity.com/paper/1610.09650/) (2016)
- [Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives](https://mlsanity.com/paper/2305.08466/) (2023)
- [Sobolev GAN](https://mlsanity.com/paper/1711.04894/) (2017)
- [Classification with Deep Neural Networks and Logistic Loss](https://mlsanity.com/paper/2307.16792/) (2023)
- [Approximation Properties of Deep ReLU CNNs](https://mlsanity.com/paper/2109.00190/) (2021)
- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [Approximation of Functionals by Neural Network without Curse of Dimensionality](https://mlsanity.com/paper/2205.14421/) (2022)
- [Nonlinear functional regression by functional deep neural network with kernel embedding](https://mlsanity.com/paper/2401.02890/) (2024)

## Built on by (most cited first, 14 of 14)

- [Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations](https://mlsanity.com/paper/2405.14099/) (2024)
- [Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms](https://mlsanity.com/paper/2508.05141/) (2025)
- [DRM Revisited: A Complete Error Analysis](https://mlsanity.com/paper/2407.09032/) (2024)
- [Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers](https://mlsanity.com/paper/2410.06308/) (2024)
- [An Imbalanced Learning-based Sampling Method for Physics-informed Neural Networks](https://mlsanity.com/paper/2501.11222/) (2025)
- [Learning Sparse Compositional Functions with Norm-Constrained Neural Networks](https://mlsanity.com/paper/2605.25608/) (2026)
- [Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equations](https://mlsanity.com/paper/2405.14096/) (2024)
- [A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks](https://mlsanity.com/paper/2412.09752/) (2024)
- [Learn Singularly Perturbed Solutions via Homotopy Dynamics](https://mlsanity.com/paper/2502.00488/) (2025)
- [Geometric regularization of autoencoders via observed stochastic dynamics](https://mlsanity.com/paper/2604.16282/) (2026)
- [DeepONet for Solving Nonlinear Partial Differential Equations with Physics-Informed Training](https://mlsanity.com/paper/2410.04344/) (2024)
- [Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learning](https://mlsanity.com/paper/2505.15064/) (2025)
- [Is data-efficient learning feasible with quantum models?](https://mlsanity.com/paper/2508.19437/) (2025)
- [On the Dimension-Free Approximation of Deep Neural Networks for Symmetric Korobov Functions](https://mlsanity.com/paper/2511.12398/) (2025)

## Abstract (arXiv)

Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question. This paper explores a comparison between deeper neural networks (DeNNs) with a flexible number of layers and wider neural networks (WeNNs) with limited hidden layers, focusing on their optimal generalization error in Sobolev losses. Analytical investigations reveal that the architecture of a neural network can be significantly influenced by various factors, including the number of sample points, parameters within the neural networks, and the regularity of the loss function. Specifically, a higher number of parameters tends to favor WeNNs, while an increased number of sample points and greater regularity in the loss function lean towards the adoption of DeNNs. We ultimately apply this theory to address partial differential equations using deep Ritz and physics-informed neural network (PINN) methods, guiding the design of neural networks.