# Speeding up and reducing memory usage for scientific machine learning via mixed precision

arXiv: https://arxiv.org/abs/2401.16645 · HTML: https://mlsanity.com/paper/2401.16645/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data](https://mlsanity.com/paper/1901.06314/) (2019)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems](https://mlsanity.com/paper/2111.02801/) (2021)
- [Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification](https://mlsanity.com/paper/1801.06879/) (2018)
- [Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems](https://mlsanity.com/paper/1809.08327/) (2018)
- [Physics-informed neural network for ultrasound nondestructive quantification of surface breaking cracks](https://mlsanity.com/paper/2005.03596/) (2020)
- [Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://mlsanity.com/paper/1807.11205/) (2018)
- [Fourier-DeepONet: Fourier-enhanced deep operator networks for full waveform inversion with improved accuracy, generalizability, and robustness](https://mlsanity.com/paper/2305.17289/) (2023)
- [Fourier-MIONet: Fourier-enhanced multiple-input neural operators for multiphase modeling of geological carbon sequestration](https://mlsanity.com/paper/2303.04778/) (2023)
- [On the Fenchel Duality between Strong Convexity and Lipschitz Continuous Gradient](https://mlsanity.com/paper/1803.06573/) (2018)
- [Systems Biology: Identifiability analysis and parameter identification via systems-biology informed neural networks](https://mlsanity.com/paper/2202.01723/) (2022)
- [Machine Learning for Partial Differential Equations](https://mlsanity.com/paper/2303.17078/) (2023)

## Built on by (most cited first, 8 of 8)

- [Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty Quantification in Deep Operator Networks](https://mlsanity.com/paper/2402.15406/) (2024)
- [Learning and discovering multiple solutions using physics-informed neural networks with random initialization and deep ensemble](https://mlsanity.com/paper/2503.06320/) (2025)
- [Efficient and generalizable nested Fourier-DeepONet for three-dimensional geological carbon sequestration](https://mlsanity.com/paper/2409.16572/) (2024)
- [Tensor-Compressed and Fully-Quantized Training of Neural PDE Solvers](https://mlsanity.com/paper/2512.09202/) (2025)
- [Mixed Precision Training of Neural ODEs](https://mlsanity.com/paper/2510.23498/) (2025)
- [Catalyzing Informed Residential Energy Retrofit Decisions via Domain-Specific LLM](https://mlsanity.com/paper/2602.20181/) (2026)
- [Curvature-aware dynamic precision approach for physics-informed neural networks](https://mlsanity.com/paper/2606.04736/) (2026)
- [Auditing Invisible Weight Updates with Reference Traces](https://mlsanity.com/paper/2607.09800/) (2026)

## Abstract (arXiv)

Scientific machine learning (SciML) has emerged as a versatile approach to address complex computational science and engineering problems. Within this field, physics-informed neural networks (PINNs) and deep operator networks (DeepONets) stand out as the leading techniques for solving partial differential equations by incorporating both physical equations and experimental data. However, training PINNs and DeepONets requires significant computational resources, including long computational times and large amounts of memory. In search of computational efficiency, training neural networks using half precision (float16) rather than the conventional single (float32) or double (float64) precision has gained substantial interest, given the inherent benefits of reduced computational time and memory consumed. However, we find that float16 cannot be applied to SciML methods, because of gradient divergence at the start of training, weight updates going to zero, and the inability to converge to a local minima. To overcome these limitations, we explore mixed precision, which is an approach that combines the float16 and float32 numerical formats to reduce memory usage and increase computational speed. Our experiments showcase that mixed precision training not only substantially decreases training times and memory demands but also maintains model accuracy. We also reinforce our empirical observations with a theoretical analysis. The research has broad implications for SciML in various computational applications.