mlsanityData to 7 Oct 2026

Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss

arXiv 2402.00152 ↗First posted 31 Jan 2024cs.LG cs.NA math.NAcited by 14

Not on the map. Why?

Abstract (arXiv)

Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question. This paper explores a comparison between deeper neural networks (DeNNs) with a flexible number of layers and wider neural networks (WeNNs) with limited hidden layers, focusing on their optimal generalization error in Sobolev losses. Analytical investigations reveal that the architecture of a neural network can be significantly influenced by various factors, including the number of sample points, parameters within the neural networks, and the regularity of the loss function. Specifically, a higher number of parameters tends to favor WeNNs, while an increased number of sample points and greater regularity in the loss function lean towards the adoption of DeNNs. We ultimately apply this theory to address partial differential equations using deep Ritz and physics-informed neural network (PINN) methods, guiding the design of neural networks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 31 of its 55 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

31 of 55 references.

References · 31

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  3. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  4. 2016Error bounds for approximations with deep ReLU networkscited by 567
  5. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  6. 2015Policy Distillationcited by 414
  7. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  8. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  9. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 188
  10. 2018Universality of Deep Convolutional Neural Networkscited by 176
  11. 2020Deep Network Approximation for Smooth Functionscited by 163
  12. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
19 more
  1. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  2. 2017Sobolev Training for Neural Networkscited by 136
  3. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  4. 2019The phase diagram of approximation rates for deep neural networkscited by 86
  5. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  6. 2018Banach Wasserstein GANcited by 56
  7. 2021Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEscited by 56
  8. 2018Generalization Error in Deep Learningcited by 46
  9. 2018Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equationscited by 44
  10. 2019Nonlinear Approximation via Compositionscited by 37
  11. 2016Deep Model Compression: Distilling Knowledge from Noisy Teacherscited by 35
  12. 2023Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivativescited by 30
  13. 2017Sobolev GANcited by 29
  14. 2023Classification with Deep Neural Networks and Logistic Losscited by 16
  15. 2021Approximation Properties of Deep ReLU CNNscited by 15
  16. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
  17. 2022Approximation of Functionals by Neural Network without Curse of Dimensionalitycited by 8
  18. 2024Nonlinear functional regression by functional deep neural network with kernel embeddingcited by 7
  19. 2023Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Normscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 2.

14 papers.

Sort:

  1. 2024Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equationscited by 9
  2. 2025Deep Neural Networks with General Activations: Super-Convergence in Sobolev Normscited by 5
  3. 2024DRM Revisited: A Complete Error Analysiscited by 4
  4. 2025An Imbalanced Learning-based Sampling Method for Physics-informed Neural Networkscited by 4
  5. 2024Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solverscited by 3
  6. 2024Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equationscited by 2
  7. 2024A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networkscited by 2
  8. 2025Learn Singularly Perturbed Solutions via Homotopy Dynamicscited by 2
  9. 2026Learning Sparse Compositional Functions with Norm-Constrained Neural Networkscited by 2
  10. 2024DeepONet for Solving Nonlinear Partial Differential Equations with Physics-Informed Trainingcited by 1
  11. 2026Geometric regularization of autoencoders via observed stochastic dynamicscited by 1
  12. 2025Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learningcited by 0
2 more
  1. 2025Is data-efficient learning feasible with quantum models?cited by 0
  2. 2025On the Dimension-Free Approximation of Deep Neural Networks for Symmetric Korobov Functionscited by 0