# Analyzing the Performance of Deep Encoder-Decoder Networks as Surrogates for a Diffusion Equation

arXiv: https://arxiv.org/abs/2302.03786 · HTML: https://mlsanity.com/paper/2302.03786/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study](https://mlsanity.com/paper/2004.03271/) (2020)
- [Machine learning for molecular simulation](https://mlsanity.com/paper/1911.02792/) (2019)
- [Scientific Machine Learning Benchmarks](https://mlsanity.com/paper/2110.12773/) (2021)
- [Fashionable Modelling with Flux](https://mlsanity.com/paper/1811.01457/) (2018)
- [Machine Learning Coarse-Grained Potentials of Protein Thermodynamics](https://mlsanity.com/paper/2212.07492/) (2022)
- [DiffEqFlux.jl - A Julia Library for Neural Differential Equations](https://mlsanity.com/paper/1902.02376/) (2019)
- [Deep Learning the Physics of Transport Phenomena](https://mlsanity.com/paper/1709.02432/) (2017)
- [Electronic excited states in deep variational Monte Carlo](https://mlsanity.com/paper/2203.09472/) (2022)
- [Weakly-Supervised Deep Learning of Heat Transport via Physics Informed Loss](https://mlsanity.com/paper/1807.11374/) (2018)
- [An unsupervised learning approach to solving heat equations on chip based on Auto Encoder and Image Gradient](https://mlsanity.com/paper/2007.09684/) (2020)
- [Learning Everywhere: A Taxonomy for the Integration of Machine Learning and Simulations](https://mlsanity.com/paper/1909.13340/) (2019)
- [Deep learning approaches to surrogates for solving the diffusion equation for mechanistic real-world simulations](https://mlsanity.com/paper/2102.05527/) (2021)

## Built on by (most cited first, 4 of 4)

- [Feasibility Study on Active Learning of Smart Surrogates for Scientific Simulations](https://mlsanity.com/paper/2407.07674/) (2024)
- [Surrogate modeling of Cellular-Potts Agent-Based Models as a segmentation task using the U-Net neural network architecture](https://mlsanity.com/paper/2505.00316/) (2025)
- [Generative diffusion model surrogates for mechanistic agent-based biological models](https://mlsanity.com/paper/2505.09630/) (2025)
- [An Active Learning-Based Streaming Pipeline for Reduced Data Training of Structure Finding Models in Neutron Diffractometry](https://mlsanity.com/paper/2506.11100/) (2025)

## Abstract (arXiv)

Neural networks (NNs) have proven to be a viable alternative to traditional direct numerical algorithms, with the potential to accelerate computational time by several orders of magnitude. In the present paper we study the use of encoder-decoder convolutional neural network (CNN) as surrogates for steady-state diffusion solvers. The construction of such surrogates requires the selection of an appropriate task, network architecture, training set structure and size, loss function, and training algorithm hyperparameters. It is well known that each of these factors can have a significant impact on the performance of the resultant model. Our approach employs an encoder-decoder CNN architecture, which we posit is particularly well-suited for this task due to its ability to effectively transform data, as opposed to merely compressing it. We systematically evaluate a range of loss functions, hyperparameters, and training set sizes. Our results indicate that increasing the size of the training set has a substantial effect on reducing performance fluctuations and overall error. Additionally, we observe that the performance of the model exhibits a logarithmic dependence on the training set size. Furthermore, we investigate the effect on model performance by using different subsets of data with varying features. Our results highlight the importance of sampling the configurational space in an optimal manner, as this can have a significant impact on the performance of the model and the required training time. In conclusion, our results suggest that training a model with a pre-determined error performance bound is not a viable approach, as it does not guarantee that edge cases with errors larger than the bound do not exist. Furthermore, as most surrogate tasks involve a high dimensional landscape, an ever increasing training set size is, in principle, needed, however it is not a practical solution.