# PottsMGNet: A Mathematical Explanation of Encoder-Decoder Based Neural Networks

arXiv: https://arxiv.org/abs/2307.09039 · HTML: https://mlsanity.com/paper/2307.09039/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 74 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 23 of 23)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation](https://mlsanity.com/paper/1912.05074/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Real Image Denoising with Feature Attention](https://mlsanity.com/paper/1904.07396/) (2019)
- [Machine Learning from a Continuous Viewpoint](https://mlsanity.com/paper/1912.12777/) (2019)
- [MgNet: A Unified Framework of Multigrid and Convolutional Neural Network](https://mlsanity.com/paper/1901.10415/) (2019)
- [Deep learning as optimal control problems: models and numerical methods](https://mlsanity.com/paper/1904.05657/) (2019)
- [Deep Convolutional Neural Networks with Spatial Regularization, Volume and Star-shape Priori for Image Segmentation](https://mlsanity.com/paper/2002.03989/) (2020)
- [Structure preserving deep learning](https://mlsanity.com/paper/2006.03364/) (2020)
- [Curvature Regularized Surface Reconstruction from Point Cloud](https://mlsanity.com/paper/2001.07884/) (2020)
- [A Color Elastica Model for Vector-Valued Image Regularization](https://mlsanity.com/paper/2008.08255/) (2020)
- [An Operator-Splitting Method for the Gaussian Curvature Regularization Model with Applications to Surface Smoothing and Imaging](https://mlsanity.com/paper/2108.01914/) (2021)
- [DOSnet as a Non-Black-Box PDE Solver: When Deep Learning Meets Operator Splitting](https://mlsanity.com/paper/2212.05571/) (2022)
- [Continuous U-Net: Faster, Greater and Noiseless](https://mlsanity.com/paper/2302.00626/) (2023)
- [New region force for variational models in image segmentation and high dimensional data clustering](https://mlsanity.com/paper/1704.08218/) (2017)
- [An efficient iterative method for reconstructing surface from point clouds](https://mlsanity.com/paper/2005.11864/) (2020)

## Built on by (most cited first, 17 of 17)

- [Double-well Net for Image Segmentation](https://mlsanity.com/paper/2401.00456/) (2023)
- [Deep Convolutional Neural Networks Meet Variational Shape Compactness Priors for Image Segmentation](https://mlsanity.com/paper/2406.19400/) (2024)
- [Euler's Elastica Based Cartoon-Smooth-Texture Image Decomposition](https://mlsanity.com/paper/2407.02794/) (2024)
- [A Mathematical Explanation of UNet](https://mlsanity.com/paper/2410.04434/) (2024)
- [EA: An Event Autoencoder for High-Speed Vision Sensing](https://mlsanity.com/paper/2507.06459/) (2025)
- [A Mathematical Explanation of Transformers](https://mlsanity.com/paper/2510.03989/) (2025)
- [Topology-Guaranteed Image Segmentation: Enforcing Connectivity, Genus, and Width Constraints](https://mlsanity.com/paper/2601.11409/) (2026)
- [Image Segmentation via Variational Model Based Tailored UNet: A Deep Variational Framework](https://mlsanity.com/paper/2505.05806/) (2025)
- [Mathematical Modeling and Convergence Analysis of Deep Neural Networks with Dense Layer Connectivities in Deep Learning](https://mlsanity.com/paper/2510.02049/) (2025)
- [LIP-CAR: contrast agent reduction by a deep learned inverse problem](https://mlsanity.com/paper/2407.10559/) (2024)
- [Robust Variational Model Based Tailored UNet: Leveraging Edge Detector and Mean Curvature for Improved Image Segmentation](https://mlsanity.com/paper/2512.07590/) (2025)
- [Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations](https://mlsanity.com/paper/2605.22557/) (2026)
- [Deep-layer limit and stability analysis of the basic forward-backward-splitting induced network (II): learning problems](https://mlsanity.com/paper/2605.27133/) (2026)
- [A Unified Variational Framework for Deep Weakly Supervised Image Segmentation](https://mlsanity.com/paper/2607.19669/) (2026)
- [LHMCF-Net: A Learned Hyperbolic Mean Curvature Flow Network for Medical Images Segmentation](https://mlsanity.com/paper/2608.20942/) (2026)
- [LiteEvent-AE: Lightweight Autoencoder for Event-Based Vision on Low-Latency Energy-Constrained Edge Devices](https://mlsanity.com/paper/2608.21764/) (2026)
- [Propagate, Then Sharpen: Post-Hoc Refinement of Frozen Node Classifiers](https://mlsanity.com/paper/2609.35080/) (2026)

## Abstract (arXiv)

For problems in image processing and many other fields, a large class of effective neural networks has encoder-decoder-based architectures. Although these networks have made impressive performances, mathematical explanations of their architectures are still underdeveloped. In this paper, we study the encoder-decoder-based network architecture from the algorithmic perspective and provide a mathematical explanation. We use the two-phase Potts model for image segmentation as an example for our explanations. We associate the segmentation problem with a control problem in the continuous setting. Then, multigrid method and operator splitting scheme, the PottsMGNet, are used to discretize the continuous control model. We show that the resulting discrete PottsMGNet is equivalent to an encoder-decoder-based network. With minor modifications, it is shown that a number of the popular encoder-decoder-based neural networks are just instances of the proposed PottsMGNet. By incorporating the Soft-Threshold-Dynamics into the PottsMGNet as a regularizer, the PottsMGNet has shown to be robust with the network parameters such as network width and depth and achieved remarkable performance on datasets with very large noise. In nearly all our experiments, the new network always performs better or as good on accuracy and dice score than existing networks for image segmentation.