# Spectrum-Informed Multistage Neural Networks: Multiscale Function Approximators of Machine Precision

arXiv: https://arxiv.org/abs/2407.17213 · HTML: https://mlsanity.com/paper/2407.17213/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains](https://mlsanity.com/paper/2007.11207/) (2020)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Multi-stage Neural Networks: Function Approximator of Machine Precision](https://mlsanity.com/paper/2307.08934/) (2023)
- [Challenges of learning multi-scale dynamics with AI weather models: Implications for stability and one solution](https://mlsanity.com/paper/2304.07029/) (2023)
- [Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow](https://mlsanity.com/paper/2206.03198/) (2022)
- [Asymptotic self-similar blow-up profile for three-dimensional axisymmetric Euler equations using neural networks](https://mlsanity.com/paper/2201.06780/) (2022)
- [Precision Machine Learning](https://mlsanity.com/paper/2210.13447/) (2022)
- [Achieving High Accuracy with PINNs via Energy Natural Gradients](https://mlsanity.com/paper/2302.13163/) (2023)
- [Ensemble flow reconstruction in the atmospheric boundary layer from spatially limited measurements through latent diffusion models](https://mlsanity.com/paper/2303.00836/) (2023)
- [PhaseDNN - A Parallel Phase Shift Deep Neural Network for Adaptive Wideband Learning](https://mlsanity.com/paper/1905.01389/) (2019)

## Built on by (most cited first, 3 of 3)

- [High precision PINNs in unbounded domains: application to singularity formulation in PDEs](https://mlsanity.com/paper/2506.19243/) (2025)
- [Separated-Variable Spectral Neural Networks: A Physics-Informed Learning Approach for High-Frequency PDEs](https://mlsanity.com/paper/2508.00628/) (2025)
- [Singularity Formation: Synergy in Theoretical, Numerical and Machine Learning Approaches](https://mlsanity.com/paper/2604.16842/) (2026)

## Abstract (arXiv)

Deep learning frameworks have become powerful tools for approaching scientific problems such as turbulent flow, which has wide-ranging applications. In practice, however, existing scientific machine learning approaches have difficulty fitting complex, multi-scale dynamical systems to very high precision, as required in scientific contexts. We propose using the novel multistage neural network approach with a spectrum-informed initialization to learn the residue from the previous stage, utilizing the spectral biases associated with neural networks to capture high frequency features in the residue, and successfully tackle the spectral bias of neural networks. This approach allows the neural network to fit target functions to double floating-point machine precision O(10-16).