mlsanityData to 14 Oct 2026

Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint

arXiv 1912.04862 ↗First posted 10 Dec 2019cs.LG cs.NA math.NAcited by 42

Not on the map. Why?

Abstract (arXiv)

Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We provide analysis of these techniques and illustrate via numerical examples dramatic increases in accuracy and convergence rate for benchmarks characterizing scientific applications where DNNs are currently used, including regression problems and physics-informed neural networks for the solution of partial differential equations.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 8; 2023: 8; 2024: 7; 2025: 7; 2026: 4.

42 papers.

Sort:

By area:

  1. 2021SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEscited by 32
  2. 2021On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elementscited by 26
  3. 2021Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Controlcited by 24
  4. 2023An Extreme Learning Machine-Based Method for Computational PDEs in Higher Dimensionscited by 21
  5. 2023Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategiescited by 20
  6. 2021Partition of unity networks: deep hp-approximationcited by 16
  7. 2023CS4ML: A general framework for active learning with arbitrary data based on Christoffel functionscited by 11
  8. 2025Anant-Net: Breaking the Curse of Dimensionality with Scalable and Interpretable Neural Surrogate for High-Dimensional PDEscited by 11
  9. 2021Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learningscited by 9
  10. 2022Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networkscited by 9
  11. 2022Numerical Computation of Partial Differential Equations by Hidden-Layer Concatenated Extreme Learning Machinecited by 9
  12. 2024Learning smooth functions in high dimensions: from sparse polynomials to deep neural networkscited by 7
30 more
  1. 2024Fourier PINNs: From Strong Boundary Conditions to Adaptive Fourier Basescited by 7
  2. 2021slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networkscited by 6
  3. 2022CAS4DL: Christoffel Adaptive Sampling for function approximation via Deep Learningcited by 6
  4. 2023Graph Neural Networks and Applied Linear Algebracited by 6
  5. 2022Sparse Deep Neural Network for Nonlinear Partial Differential Equationscited by 5
  6. 2023Training-Free Neural Active Learning with Initialization-Robustness Guaranteescited by 5
  7. 2022Solving Elliptic Problems with Singular Sources using Singularity Splitting Deep Ritz Methodcited by 4
  8. 2024A Structure-Guided Gauss-Newton Method for Shallow ReLU Neural Networkcited by 3
  9. 2025Deep Operator Neural Network Model Predictive Controlcited by 3
  10. 2025Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processescited by 3
  11. 2021Probabilistic partition of unity networks: clustering based deep approximationcited by 2
  12. 2022NINNs: Nudging Induced Neural Networkscited by 2
  13. 2022Probabilistic partition of unity networks for high-dimensional regression problemscited by 2
  14. 2025Hybrid Least Squares/Gradient Descent Methods for DeepONetscited by 2
  15. 2023Error Analysis of Physics-Informed Neural Networks for Approximating Dynamic PDEs of Second Order in Timecited by 1
  16. 2023A Note on Dimensionality Reduction in Deep Neural Networks using Empirical Interpolation Methodcited by 1
  17. 2024Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximationcited by 1
  18. 2026Extending Fourier Neural Operators for Modeling Parameterized and Coupled PDEscited by 1
  19. 2021Polynomial-Spline Neural Networks with Exact Integralscited by 0
  20. 2022Anisotropic, Sparse and Interpretable Physics-Informed Neural Networks for PDEscited by 0
  21. 2023On the accuracy of interpolation based on single-layer artificial neural networks with a focus on defeating the Runge phenomenoncited by 0
  22. 2024Error Analysis and Numerical Algorithm for PDE Approximation with Hidden-Layer Concatenated Physics Informed Neural Networkscited by 0
  23. 2024Efficient Shallow Ritz Method For 1D Diffusion-Reaction Problemscited by 0
  24. 2024Gaussian Variational Schemes on Bounded and Unbounded Domainscited by 0
  25. 2025Approach to Finding a Robust Deep Learning Modelcited by 0
  26. 2025A Saddle Point Remedy: Power of Variable Elimination in Non-convex Optimizationcited by 0
  27. 2025SUPN: Shallow Universal Polynomial Networks for Efficient Function Approximationcited by 0
  28. 2026A Green-Integral-Constrained Neural Solver with Stochastic Physics-Informed Regularizationcited by 0
  29. 2026Hybrid Least Squares/Gradient Descent Methods for MIONetscited by 0
  30. 2026Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interestcited by 0