# Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint

arXiv: https://arxiv.org/abs/1912.04862 · HTML: https://mlsanity.com/paper/1912.04862/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2019.

## Built on by (most cited first, 30 of 42)

- [SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs](https://mlsanity.com/paper/2102.13037/) (2021)
- [On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements](https://mlsanity.com/paper/2110.14121/) (2021)
- [Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Control](https://mlsanity.com/paper/2105.14094/) (2021)
- [An Extreme Learning Machine-Based Method for Computational PDEs in Higher Dimensions](https://mlsanity.com/paper/2309.07049/) (2023)
- [Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies](https://mlsanity.com/paper/2306.17648/) (2023)
- [Partition of unity networks: deep hp-approximation](https://mlsanity.com/paper/2101.11256/) (2021)
- [CS4ML: A general framework for active learning with arbitrary data based on Christoffel functions](https://mlsanity.com/paper/2306.00945/) (2023)
- [Anant-Net: Breaking the Curse of Dimensionality with Scalable and Interpretable Neural Surrogate for High-Dimensional PDEs](https://mlsanity.com/paper/2505.03595/) (2025)
- [Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings](https://mlsanity.com/paper/2107.11496/) (2021)
- [Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networks](https://mlsanity.com/paper/2201.09989/) (2022)
- [Numerical Computation of Partial Differential Equations by Hidden-Layer Concatenated Extreme Learning Machine](https://mlsanity.com/paper/2204.11375/) (2022)
- [Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks](https://mlsanity.com/paper/2404.03761/) (2024)
- [Fourier PINNs: From Strong Boundary Conditions to Adaptive Fourier Bases](https://mlsanity.com/paper/2410.03496/) (2024)
- [slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks](https://mlsanity.com/paper/2109.14002/) (2021)
- [CAS4DL: Christoffel Adaptive Sampling for function approximation via Deep Learning](https://mlsanity.com/paper/2208.12190/) (2022)
- [Graph Neural Networks and Applied Linear Algebra](https://mlsanity.com/paper/2310.14084/) (2023)
- [Sparse Deep Neural Network for Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2207.13266/) (2022)
- [Training-Free Neural Active Learning with Initialization-Robustness Guarantees](https://mlsanity.com/paper/2306.04454/) (2023)
- [Solving Elliptic Problems with Singular Sources using Singularity Splitting Deep Ritz Method](https://mlsanity.com/paper/2209.02931/) (2022)
- [A Structure-Guided Gauss-Newton Method for Shallow ReLU Neural Network](https://mlsanity.com/paper/2404.05064/) (2024)
- [Deep Operator Neural Network Model Predictive Control](https://mlsanity.com/paper/2505.18008/) (2025)
- [Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes](https://mlsanity.com/paper/2510.05568/) (2025)
- [Probabilistic partition of unity networks: clustering based deep approximation](https://mlsanity.com/paper/2107.03066/) (2021)
- [NINNs: Nudging Induced Neural Networks](https://mlsanity.com/paper/2203.07947/) (2022)
- [Probabilistic partition of unity networks for high-dimensional regression problems](https://mlsanity.com/paper/2210.02694/) (2022)
- [Hybrid Least Squares/Gradient Descent Methods for DeepONets](https://mlsanity.com/paper/2508.15394/) (2025)
- [Error Analysis of Physics-Informed Neural Networks for Approximating Dynamic PDEs of Second Order in Time](https://mlsanity.com/paper/2303.12245/) (2023)
- [A Note on Dimensionality Reduction in Deep Neural Networks using Empirical Interpolation Method](https://mlsanity.com/paper/2305.09842/) (2023)
- [Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation](https://mlsanity.com/paper/2406.02336/) (2024)
- [Extending Fourier Neural Operators for Modeling Parameterized and Coupled PDEs](https://mlsanity.com/paper/2607.23466/) (2026)

## Abstract (arXiv)

Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We provide analysis of these techniques and illustrate via numerical examples dramatic increases in accuracy and convergence rate for benchmarks characterizing scientific applications where DNNs are currently used, including regression problems and physics-informed neural networks for the solution of partial differential equations.