# Translating Numerical Concepts for PDEs into Neural Architectures

arXiv: https://arxiv.org/abs/2103.15419 · HTML: https://mlsanity.com/paper/2103.15419/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration](https://mlsanity.com/paper/1508.02848/) (2015)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Learning to Optimize Multigrid PDE Solvers](https://mlsanity.com/paper/1902.10248/) (2019)
- [PDE-based Group Equivariant Convolutional Neural Networks](https://mlsanity.com/paper/2001.09046/) (2020)
- [Forward Stability of ResNet and Its Variants](https://mlsanity.com/paper/1811.09885/) (2018)
- [Lifting Layers: Analysis and Applications](https://mlsanity.com/paper/1803.08660/) (2018)
- [Translating Diffusion, Wavelets, and Regularisation into Residual Networks](https://mlsanity.com/paper/2002.02753/) (2020)

## Built on by (most cited first, 4 of 4)

- [Connections between Numerical Algorithms for PDEs and Neural Networks](https://mlsanity.com/paper/2107.14742/) (2021)
- [Distributed Multigrid Neural Solvers on Megavoxel Domains](https://mlsanity.com/paper/2104.14538/) (2021)
- [Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods](https://mlsanity.com/paper/2108.13993/) (2021)
- [Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations](https://mlsanity.com/paper/2109.00095/) (2021)

## Abstract (arXiv)

We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent networks, and U-nets. These connections guarantee Euclidean stability of specific ResNets with a transposed convolution layer structure in each block. We present three numerical justifications for skip connections: as time discretisations in explicit schemes, as extrapolation mechanisms for accelerating those methods, and as recurrent connections in fixed point solvers for implicit schemes. Last but not least, we also motivate uncommon design choices such as nonmonotone activation functions. Our findings give a numerical perspective on the success of modern neural network architectures, and they provide design criteria for stable networks.