# Multi-level Residual Networks from Dynamical Systems View

arXiv: https://arxiv.org/abs/1710.10348 · HTML: https://mlsanity.com/paper/1710.10348/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2017.

## Built on by (most cited first, 30 of 59)

- [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [Encoding physics to learn reaction-diffusion processes](https://mlsanity.com/paper/2106.04781/) (2021)
- [1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities](https://mlsanity.com/paper/2503.14858/) (2025)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021)
- [Constructing Neural Network-Based Models for Simulating Dynamical Systems](https://mlsanity.com/paper/2111.01495/) (2021)
- [Neural Delay Differential Equations](https://mlsanity.com/paper/2102.10801/) (2021)
- [The Elastic Lottery Ticket Hypothesis](https://mlsanity.com/paper/2103.16547/) (2021)
- [LEMON: Lossless model expansion](https://mlsanity.com/paper/2310.07999/) (2023)
- [Learning optimal multigrid smoothers via neural networks](https://mlsanity.com/paper/2102.12071/) (2021)
- [Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules](https://mlsanity.com/paper/2206.01649/) (2022)
- [Stateful ODE-Nets using Basis Function Expansions](https://mlsanity.com/paper/2106.10820/) (2021)
- [Globally Convergent Multilevel Training of Deep Residual Networks](https://mlsanity.com/paper/2107.07572/) (2021)
- [ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models](https://mlsanity.com/paper/2602.17951/) (2026)
- [Preparing Lessons for Progressive Training on Language Models](https://mlsanity.com/paper/2401.09192/) (2024)
- [Neural equilibria for long-term prediction of nonlinear conservation laws](https://mlsanity.com/paper/2501.06933/) (2025)
- [Understanding Self-attention Mechanism via Dynamical System Perspective](https://mlsanity.com/paper/2308.09939/) (2023)
- [Interplay between depth and width for interpolation in neural ODEs](https://mlsanity.com/paper/2401.09902/) (2024)
- [Hard Encoding of Physics for Learning Spatiotemporal Dynamics](https://mlsanity.com/paper/2105.00557/) (2021)
- [A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms](https://mlsanity.com/paper/2211.12386/) (2022)
- [Speeding up Deep Model Training by Sharing Weights and Then Unsharing](https://mlsanity.com/paper/2110.03848/) (2021)
- [NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clients](https://mlsanity.com/paper/2308.07761/) (2023)
- [An Interpretive Constrained Linear Model for ResNet and MgNet](https://mlsanity.com/paper/2112.07441/) (2021)
- [Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training](https://mlsanity.com/paper/2302.07049/) (2023)
- [HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMs](https://mlsanity.com/paper/2506.00088/) (2025)
- [Efficient Training of Large Vision Models via Advanced Automated Progressive Learning](https://mlsanity.com/paper/2410.00350/) (2024)
- [LESA: Learnable LLM Layer Scaling-Up](https://mlsanity.com/paper/2502.13794/) (2025)
- [Multiscale Invertible Generative Networks for High-Dimensional Bayesian Inference](https://mlsanity.com/paper/2105.05489/) (2021)
- [Neural Piecewise-Constant Delay Differential Equations](https://mlsanity.com/paper/2201.00960/) (2022)

## Abstract (arXiv)

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet theoretically, such as unraveled view, unrolled iterative estimation and dynamical systems view. In this paper, we adopt the dynamical systems point of view, and analyze the lesioning properties of ResNet both theoretically and experimentally. Based on these analyses, we additionally propose a novel method for accelerating ResNet training. We apply the proposed method to train ResNets and Wide ResNets for three image classification benchmarks, reducing training time by more than 40% with superior or on-par accuracy.