Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Bias
Not on the map. Why?
Abstract (arXiv)
Recent advances in machine learning have inspired a surge of research into reconstructing specific quantities of interest from measurements that comply with certain physical laws. These efforts focus on inverse problems that are governed by partial differential equations (PDEs). In this work, we develop an asymptotic Sobolev norm learning curve for kernel ridge(less) regression when addressing (elliptical) linear inverse problems. Our results show that the PDE operators in the inverse problem can stabilize the variance and even behave benign overfitting for fixed-dimensional problems, exhibiting different behaviors from regression problems. Besides, our investigation also demonstrates the impact of various inductive biases introduced by minimizing different Sobolev norms as a form of implicit regularization. For the regularized least squares estimator, we find that all considered inductive biases can achieve the optimal convergence rate, provided the regularization parameter is appropriately chosen. The convergence rate is actually independent to the choice of (smooth enough) inductive bias for both ridge and ridgeless regression. Surprisingly, our smoothness requirement recovered the condition found in Bayesian setting and extend the conclusion to the minimum norm interpolation estimators.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 88 references that are in the map. First posted on arXiv 13 Jun 2024.
Builds on
46 of 88 references.
References · 46
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
- 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
- 2017Solving high-dimensional partial differential equations using deep learningcited by 549
- 2015An Introduction to Matrix Concentration Inequalitiescited by 391
- 2017PDE-Net: Learning PDEs from Datacited by 304
- 2021Gradient-enhanced physics-informed neural networks for forward and inverse PDE problemscited by 220
- 2018PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Networkcited by 208
- 2021Solving and Learning Nonlinear PDEs with Gaussian Processescited by 128
34 more
- 2023An Expert's Guide to Training Physics-informed Neural Networkscited by 119
- 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
- 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
- 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
- 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 78
- 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
- 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
- 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
- 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
- 2021Plugin Estimation of Smooth Optimal Transport Mapscited by 67
- 2021Convergence Rates for Learning Linear Operators from Noisy Datacited by 55
- 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
- 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
- 2020Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theorycited by 41
- 2021Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimalitycited by 40
- 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
- 2020Deep neural network approximation for high-dimensional elliptic PDEs with boundary conditionscited by 25
- 2019A unifying representer theorem for inverse problems and machine learningcited by 24
- 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
- 2020Inference of dynamic systems from noisy and sparse data via manifold-constrained Gaussian processescited by 18
- 2021A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equationscited by 18
- 2021Parametric Complexity Bounds for Approximating PDEs with Neural Networkscited by 18
- 2016Convergence guarantees for kernel-based quadrature rules in misspecified settingscited by 15
- 2020Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learningcited by 14
- 2021Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networkscited by 14
- 2022Minimax Optimal Kernel Operator Learning via Multilevel Trainingcited by 13
- 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
- 2022Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descentcited by 11
- 2024Physics-informed machine learning as a kernel methodcited by 11
- 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
- 2021Manifold-constrained Gaussian process inference for time-varying parameters in dynamic systemscited by 8
- 2019CURE: Curvature Regularization For Missing Data Recoverycited by 7
- 2023Variational Gaussian Processes For Linear Inverse Problemscited by 2
- 2024Spectrally Transformed Kernel Regressioncited by 2
Built on by
Not cited yet.