# Multi-Grade Deep Learning

arXiv: https://arxiv.org/abs/2302.00150 · HTML: https://mlsanity.com/paper/2302.00150/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 1 Feb 2023.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1801.06637/) (2018)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Nonlinear Approximation and (Deep) ReLU Networks](https://mlsanity.com/paper/1905.02199/) (2019)
- [DeepPET: A deep encoder-decoder network for directly solving the PET reconstruction inverse problem](https://mlsanity.com/paper/1804.07851/) (2018)
- [Convergence of Deep ReLU Networks](https://mlsanity.com/paper/2107.12530/) (2021)
- [Convergence of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2109.13542/) (2021)
- [Sparse Deep Neural Network for Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2207.13266/) (2022)
- [Parameter Choices for Sparse Regularization with the ℓ1 Norm](https://mlsanity.com/paper/2205.07170/) (2022)

## Built on by (most cited first, 12 of 12)

- [Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning](https://mlsanity.com/paper/2410.16105/) (2024)
- [Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths](https://mlsanity.com/paper/2306.01692/) (2023)
- [Multi-Grade Deep Learning for Partial Differential Equations with Applications to the Burgers Equation](https://mlsanity.com/paper/2309.07401/) (2023)
- [Deep Neural Network Solutions for Oscillatory Fredholm Integral Equations](https://mlsanity.com/paper/2401.07003/) (2024)
- [Successive Affine Learning for Deep Neural Networks](https://mlsanity.com/paper/2305.07996/) (2023)
- [Hypothesis Spaces for Deep Learning](https://mlsanity.com/paper/2403.03353/) (2024)
- [No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks](https://mlsanity.com/paper/2405.02369/) (2024)
- [Computational Advantages of Multi-Grade Deep Learning: Convergence Analysis and Performance Insights](https://mlsanity.com/paper/2507.20351/) (2025)
- [Multigrade Neural Network Approximation](https://mlsanity.com/paper/2601.16884/) (2026)
- [Layer-wise Geometric Approximation Rates for Deep Networks](https://mlsanity.com/paper/2604.20219/) (2026)
- [MG-SpaIR: Multi-grade Sparse-guided Implicit Representation for Training-Data-Free Image Restoration](https://mlsanity.com/paper/2607.00138/) (2026)
- [Multi-stage neural operator learning with application for convolutions](https://mlsanity.com/paper/2608.18851/) (2026)

## Abstract (arXiv)

The current deep learning model is of a single-grade, that is, it learns a deep neural network by solving a single nonconvex optimization problem. When the layer number of the neural network is large, it is computationally challenging to carry out such a task efficiently. Inspired by the human education process which arranges learning in grades, we propose a multi-grade learning model: We successively solve a number of optimization problems of small sizes, which are organized in grades, to learn a shallow neural network for each grade. Specifically, the current grade is to learn the leftover from the previous grade. In each of the grades, we learn a shallow neural network stacked on the top of the neural network, learned in the previous grades, which remains unchanged in training of the current and future grades. By dividing the task of learning a deep neural network into learning several shallow neural networks, one can alleviate the severity of the nonconvexity of the original optimization problem of a large size. When all grades of the learning are completed, the final neural network learned is a stair-shape neural network, which is the superposition of networks learned from all grades. Such a model enables us to learn a deep neural network much more effectively and efficiently. Moreover, multi-grade learning naturally leads to adaptive learning. We prove that in the context of function approximation if the neural network generated by a new grade is nontrivial, the optimal error of the grade is strictly reduced from the optimal error of the previous grade. Furthermore, we provide several proof-of-concept numerical examples which demonstrate that the proposed multi-grade model outperforms significantly the traditional single-grade model and is much more robust than the traditional model.