# Exascale Deep Learning for Scientific Inverse Problems

arXiv: https://arxiv.org/abs/1909.11150 · HTML: https://mlsanity.com/paper/1909.11150/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2019.

## Built on by (most cited first, 3 of 3)

- [Applications of Machine Learning to Lattice Quantum Field Theory](https://mlsanity.com/paper/2202.05838/) (2022)
- [JUWELS Booster -- A Supercomputer for Large-Scale AI Research](https://mlsanity.com/paper/2108.11976/) (2021)
- [Neural Networks Asymptotic Behaviours for the Resolution of Inverse Problems](https://mlsanity.com/paper/2402.09338/) (2024)

## Abstract (arXiv)

We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping of gradient tensors. These new techniques produce an optimal overlap between computation and communication and result in near-linear scaling (0.93) of distributed training up to 27,600 NVIDIA V100 GPUs on the Summit Supercomputer. We demonstrate our gradient reduction techniques in the context of training a Fully Convolutional Neural Network to approximate the solution of a longstanding scientific inverse problem in materials imaging. The efficient distributed training on a dataset size of 0.5 PB, produces a model capable of an atomically-accurate reconstruction of materials, and in the process reaching a peak performance of 2.15(4) EFLOPS16.