# Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives

arXiv: https://arxiv.org/abs/2305.08466 · HTML: https://mlsanity.com/paper/2305.08466/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 18 of 18)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Towards Deep Neural Network Architectures Robust to Adversarial Examples](https://mlsanity.com/paper/1412.5068/) (2014)
- [Sobolev Training for Neural Networks](https://mlsanity.com/paper/1706.04859/) (2017)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [Banach Wasserstein GAN](https://mlsanity.com/paper/1806.06621/) (2018)
- [Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces](https://mlsanity.com/paper/2201.00217/) (2022)
- [Nonlinear Approximation via Compositions](https://mlsanity.com/paper/1902.10170/) (2019)
- [Deep Model Compression: Distilling Knowledge from Noisy Teachers](https://mlsanity.com/paper/1610.09650/) (2016)
- [Sobolev GAN](https://mlsanity.com/paper/1711.04894/) (2017)
- [Error Estimates for the Deep Ritz Method with Boundary Penalty](https://mlsanity.com/paper/2103.01007/) (2021)

## Built on by (most cited first, 30 of 30)

- [Convergence of Continuous Normalizing Flows for Learning Probability Distributions](https://mlsanity.com/paper/2404.00551/) (2024)
- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations](https://mlsanity.com/paper/2405.14099/) (2024)
- [Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensions](https://mlsanity.com/paper/2312.14276/) (2023)
- [Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms](https://mlsanity.com/paper/2508.05141/) (2025)
- [Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2309.15244/) (2023)
- [Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Norms](https://mlsanity.com/paper/2311.04779/) (2023)
- [Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers](https://mlsanity.com/paper/2410.06308/) (2024)
- [An Imbalanced Learning-based Sampling Method for Physics-informed Neural Networks](https://mlsanity.com/paper/2501.11222/) (2025)
- [RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessment](https://mlsanity.com/paper/2506.21037/) (2025)
- [Some Super-approximation Rates of ReLU Neural Networks for Korobov Functions](https://mlsanity.com/paper/2507.10345/) (2025)
- [Learning Sparse Compositional Functions with Norm-Constrained Neural Networks](https://mlsanity.com/paper/2605.25608/) (2026)
- [Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equations](https://mlsanity.com/paper/2405.14096/) (2024)
- [Expressive Power of Deep Networks on Manifolds: Simultaneous Approximation](https://mlsanity.com/paper/2509.09362/) (2025)
- [Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method](https://mlsanity.com/paper/2405.11451/) (2024)
- [Blessing of Dimensionality for Approximating Sobolev Classes on Manifolds](https://mlsanity.com/paper/2408.06996/) (2024)
- [DeepONet for Solving Nonlinear Partial Differential Equations with Physics-Informed Training](https://mlsanity.com/paper/2410.04344/) (2024)
- [Curse of Dimensionality in Neural Network Optimization](https://mlsanity.com/paper/2502.05360/) (2025)
- [Statistical Learning Guarantees for Group-Invariant Barron Functions](https://mlsanity.com/paper/2509.23474/) (2025)
- [A Quantitative Approximation Framework for Flow Distillation in Diffusion Models](https://mlsanity.com/paper/2606.03820/) (2026)
- [Generalization bounds for regression and classification on adaptive covering input domains](https://mlsanity.com/paper/2407.19715/) (2024)
- [Generalizability of Neural Networks Minimizing Empirical Risk Based on Expressive Ability](https://mlsanity.com/paper/2503.04111/) (2025)
- [Energy Approach from varepsilon-Graph to Continuum Diffusion Model with Connectivity Functional](https://mlsanity.com/paper/2510.25114/) (2025)
- [On the Dimension-Free Approximation of Deep Neural Networks for Symmetric Korobov Functions](https://mlsanity.com/paper/2511.12398/) (2025)
- [Understanding Generalization through Decision Pattern Shift](https://mlsanity.com/paper/2605.13148/) (2026)
- [A Rigorous, Tractable Measure of Model Complexity](https://mlsanity.com/paper/2605.21167/) (2026)
- [Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations](https://mlsanity.com/paper/2605.22557/) (2026)
- [Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width](https://mlsanity.com/paper/2607.10589/) (2026)
- [Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks](https://mlsanity.com/paper/2609.19937/) (2026)

## Abstract (arXiv)

This paper addresses the problem of nearly optimal Vapnik--Chervonenkis dimension (VC-dimension) and pseudo-dimension estimations of the derivative functions of deep neural networks (DNNs). Two important applications of these estimations include: 1) Establishing a nearly tight approximation result of DNNs in the Sobolev space; 2) Characterizing the generalization error of machine learning methods with loss functions involving function derivatives. This theoretical investigation fills the gap of learning error estimations for a wide range of physics-informed machine learning models and applications including generative models, solving partial differential equations, operator learning, network compression, distillation, regularization, etc.