# Asynchronous Execution of Heterogeneous Tasks in ML-driven HPC Workflows

arXiv: https://arxiv.org/abs/2208.11069 · HTML: https://mlsanity.com/paper/2208.11069/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 23 Aug 2022.

## Builds on (most cited first, 9 of 9)

- [Pathways: Asynchronous Distributed Dataflow for ML](https://mlsanity.com/paper/2203.12533/) (2022)
- [DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding](https://mlsanity.com/paper/1909.07817/) (2019)
- [Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing](https://mlsanity.com/paper/2110.02827/) (2021)
- [Coupling streaming AI and HPC ensembles to achieve 100-1000x faster biomolecular simulations](https://mlsanity.com/paper/2104.04797/) (2021)
- [Enabling Machine Learning-Ready HPC Ensembles with Merlin](https://mlsanity.com/paper/1912.02892/) (2019)
- [Pandemic Drugs at Pandemic Speed: Infrastructure for Accelerating COVID-19 Drug Discovery with Hybrid Machine Learning- and Physics-based Simulations on High Performance Computers](https://mlsanity.com/paper/2103.02843/) (2021)
- [IMPECCABLE: Integrated Modeling PipelinE for COVID Cure by Assessing Better LEads](https://mlsanity.com/paper/2010.06574/) (2020)
- [Harnessing the Power of Many: Extensible Toolkit for Scalable Ensemble Applications](https://mlsanity.com/paper/1710.08491/) (2017)
- [Ensemble Toolkit: Scalable and Flexible Execution of Ensembles of Tasks](https://mlsanity.com/paper/1602.00678/) (2016)

## Abstract (arXiv)

Heterogeneous scientific workflows consist of numerous types of tasks that require executing on heterogeneous resources. Asynchronous execution of those tasks is crucial to improve resource utilization, task throughput and reduce workflows' makespan. Therefore, middleware capable of scheduling and executing different task types across heterogeneous resources must enable asynchronous execution of tasks. In this paper, we investigate the requirements and properties of the asynchronous task execution of machine learning (ML)-driven high performance computing (HPC) workflows. We model the degree of asynchronicity permitted for arbitrary workflows and propose key metrics that can be used to determine qualitative benefits when employing asynchronous execution. Our experiments represent relevant scientific drivers, we perform them at scale on Summit, and we show that the performance enhancements due to asynchronous execution are consistent with our model.