# Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning

arXiv: https://arxiv.org/abs/2410.16105 · HTML: https://mlsanity.com/paper/2410.16105/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Towards Understanding the Spectral Bias of Deep Learning](https://mlsanity.com/paper/1912.01198/) (2019)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.05830/) (2015)
- [Deep Learning for Financial Applications : A Survey](https://mlsanity.com/paper/2002.05786/) (2020)
- [Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains](https://mlsanity.com/paper/2007.11207/) (2020)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Deep vs. shallow networks : An approximation theory perspective](https://mlsanity.com/paper/1608.03287/) (2016)
- [Understanding training and generalization in deep learning by Fourier analysis](https://mlsanity.com/paper/1808.04295/) (2018)
- [An error analysis of generative adversarial networks for learning distributions](https://mlsanity.com/paper/2105.13010/) (2021)
- [A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problems](https://mlsanity.com/paper/1909.11759/) (2019)
- [Multi-scale Deep Neural Networks for Solving High Dimensional PDEs](https://mlsanity.com/paper/1910.11710/) (2019)
- [Convergence of Deep ReLU Networks](https://mlsanity.com/paper/2107.12530/) (2021)
- [Convergence of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2109.13542/) (2021)
- [Multi-Grade Deep Learning](https://mlsanity.com/paper/2302.00150/) (2023)
- [Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths](https://mlsanity.com/paper/2306.01692/) (2023)
- [Deep Neural Network Solutions for Oscillatory Fredholm Integral Equations](https://mlsanity.com/paper/2401.07003/) (2024)
- [Inexact FPPA for the ℓ0 Sparse Regularization Problem](https://mlsanity.com/paper/2404.17689/) (2024)

## Built on by (most cited first, 14 of 14)

- [Turb-L1: Achieving Long-term Turbulence Tracing By Tackling Spectral Bias](https://mlsanity.com/paper/2505.19038/) (2025)
- [Spectral Bias in Variational Quantum Machine Learning](https://mlsanity.com/paper/2506.22555/) (2025)
- [MGT-Prism: Enhancing Domain Generalization for Machine-Generated Text Detection via Spectral Alignment](https://mlsanity.com/paper/2508.13768/) (2025)
- [Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential](https://mlsanity.com/paper/2502.18959/) (2025)
- [Computational Advantages of Multi-Grade Deep Learning: Convergence Analysis and Performance Insights](https://mlsanity.com/paper/2507.20351/) (2025)
- [Low-Rank Key Value Attention](https://mlsanity.com/paper/2601.11471/) (2026)
- [Multigrade Neural Network Approximation](https://mlsanity.com/paper/2601.16884/) (2026)
- [Layer-wise Geometric Approximation Rates for Deep Networks](https://mlsanity.com/paper/2604.20219/) (2026)
- [On the accuracy of implicit neural representations for cardiovascular anatomies and hemodynamic fields](https://mlsanity.com/paper/2510.20970/) (2025)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [OLMA: One Loss for More Accurate Time Series Forecasting](https://mlsanity.com/paper/2505.11567/) (2025)
- [SUPER Module for Detail-Sensitive and Cost-Efficient U-Net Variant Decoders](https://mlsanity.com/paper/2511.11015/) (2025)
- [CFNN: Continued Fraction Neural Network](https://mlsanity.com/paper/2603.20634/) (2026)
- [MG-SpaIR: Multi-grade Sparse-guided Implicit Representation for Training-Data-Free Image Restoration](https://mlsanity.com/paper/2607.00138/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) suffer from the spectral bias, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN composed with the SNNs trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs.