High Throughput Training of Deep Surrogates from Large Ensemble Runs
Not on the map. Why?
Abstract (arXiv)
Recent years have seen a surge in deep learning approaches to accelerate numerical solvers, which provide faithful but computationally intensive simulations of the physical world. These deep surrogates are generally trained in a supervised manner from limited amounts of data slowly generated by the same solver they intend to accelerate. We propose an open-source framework that enables the online training of these models from a large ensemble run of simulations. It leverages multiple levels of parallelism to generate rich datasets. The framework avoids I/O bottlenecks and storage issues by directly streaming the generated data. A training reservoir mitigates the inherent bias of streaming while maximizing GPU throughput. Experiment on training a fully connected network as a surrogate for the heat equation shows the proposed approach enables training on 8TB of data in 2 hours with an accuracy improved by 47% and a batch throughput multiplied by 13 compared to a traditional offline procedure.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 53 of its 103 references that are in the map. First posted on arXiv 28 Sep 2023.
Builds on
53 of 103 references.
References · 53
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2021E(n) Equivariant Graph Neural Networkscited by 951
- 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
- 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
38 more
- 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
- 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
- 2019The frontier of simulation-based inferencecited by 490
- 2021Machine learning accelerated computational fluid dynamicscited by 461
- 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
- 2019Machine Learning for Fluid Mechanicscited by 380
- 2022Message Passing Neural PDE Solverscited by 368
- 2020Universal Differential Equations for Scientific Machine Learningcited by 345
- 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
- 2019Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Datacited by 261
- 2018Distributed Prioritized Experience Replaycited by 260
- 2018Online Learning: A Comprehensive Surveycited by 203
- 2019Towards Physics-informed Deep Learning for Turbulent Flow Predictioncited by 203
- 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
- 2020Revisiting Fundamentals of Experience Replaycited by 171
- 2018Deep Fluids: A Generative Network for Parameterized Fluid Simulationscited by 121
- 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 87
- 2017Online Deep Learning: Learning Deep Neural Networks on the Flycited by 86
- 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
- 2021Simulation Intelligence: Towards a New Generation of Scientific Methodscited by 68
- 2019A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problemscited by 59
- 2020NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation frameworkcited by 58
- 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 58
- 2019Experience Replay Optimizationcited by 47
- 2019Parsl: Pervasive Parallel Programming in Pythoncited by 36
- 2020Building high accuracy emulators for scientific simulations with deep neural architecture searchcited by 36
- 2020High Performance I/O For Large Scale Deep Learningcited by 25
- 2019Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scalecited by 18
- 2019On the Acceleration of Deep Learning Model Parallelism with Stalenesscited by 17
- 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
- 2019DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Foldingcited by 15
- 2021Clairvoyant Prefetching for Distributed Machine Learning I/Ocited by 14
- 2021Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computingcited by 10
- 2021Coupling streaming AI and HPC ensembles to achieve 100-1000x faster biomolecular simulationscited by 8
- 2019RADICAL-Cybertools: Middleware Building Blocks for Scalable Sciencecited by 6
- 2018Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flashcited by 5
- 2023Training Deep Surrogate Models with Large Scale Online Learningcited by 5
- 2020Korali: Efficient and Scalable Software Framework for Bayesian Uncertainty Quantification and Stochastic Optimizationcited by 4
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.