# Optimization Algorithm Inspired Deep Neural Network Structure Design

arXiv: https://arxiv.org/abs/1810.01638 · HTML: https://mlsanity.com/paper/1810.01638/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2018.

## Built on by (most cited first, 14 of 14)

- [Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?](https://mlsanity.com/paper/2203.09081/) (2022)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [Momentum Residual Neural Networks](https://mlsanity.com/paper/2102.07870/) (2021)
- [Improving Transformers with Probabilistic Attention Keys](https://mlsanity.com/paper/2110.08678/) (2021)
- [Towards Theoretically Inspired Neural Initialization Optimization](https://mlsanity.com/paper/2210.05956/) (2022)
- [How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies](https://mlsanity.com/paper/2110.07034/) (2021)
- [MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts](https://mlsanity.com/paper/2410.14574/) (2024)
- [An Interpretive Constrained Linear Model for ResNet and MgNet](https://mlsanity.com/paper/2112.07441/) (2021)
- [Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization](https://mlsanity.com/paper/2208.00579/) (2022)
- [Diffusion Sampling with Momentum for Mitigating Divergence Artifacts](https://mlsanity.com/paper/2307.11118/) (2023)
- [Online Attentive Kernel-Based Temporal Difference Learning](https://mlsanity.com/paper/2201.09065/) (2022)
- [A Note on Machine Learning Approach for Computational Imaging](https://mlsanity.com/paper/2202.11883/) (2022)
- [Dual-Frequency Filtering Self-aware Graph Neural Networks for Homophilic and Heterophilic Graphs](https://mlsanity.com/paper/2411.11284/) (2024)
- [Looped Transformers as Optimizers](https://mlsanity.com/paper/2609.37379/) (2026)

## Abstract (arXiv)

Deep neural networks have been one of the dominant machine learning approaches in recent years. Several new network structures are proposed and have better performance than the traditional feedforward neural network structure. Representative ones include the skip connection structure in ResNet and the dense connection structure in DenseNet. However, it still lacks a unified guidance for the neural network structure design. In this paper, we propose the hypothesis that the neural network structure design can be inspired by optimization algorithms and a faster optimization algorithm may lead to a better neural network structure. Specifically, we prove that the propagation in the feedforward neural network with the same linear transformation in different layers is equivalent to minimizing some function using the gradient descent algorithm. Based on this observation, we replace the gradient descent algorithm with the heavy ball algorithm and Nesterov's accelerated gradient descent algorithm, which are faster and inspire us to design new and better network structures. ResNet and DenseNet can be considered as two special cases of our framework. Numerical experiments on CIFAR-10, CIFAR-100 and ImageNet verify the advantage of our optimization algorithm inspired structures over ResNet and DenseNet.