# Employing Artificial Intelligence to Steer Exascale Workflows with Colmena

arXiv: https://arxiv.org/abs/2408.14434 · HTML: https://mlsanity.com/paper/2408.14434/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 73 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Parsl: Pervasive Parallel Programming in Python](https://mlsanity.com/paper/1905.02158/) (2019)
- [funcX: A Federated Function Serving Fabric for Science](https://mlsanity.com/paper/2005.04215/) (2020)
- [AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification](https://mlsanity.com/paper/2110.13511/) (2021)
- [Message-passing neural networks for high-throughput polymer screening](https://mlsanity.com/paper/1807.10363/) (2018)
- [DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding](https://mlsanity.com/paper/1909.07817/) (2019)
- [Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing](https://mlsanity.com/paper/2110.02827/) (2021)
- [A generative artificial intelligence framework based on a molecular diffusion model for the design of metal-organic frameworks for carbon capture](https://mlsanity.com/paper/2306.08695/) (2023)
- [Coupling streaming AI and HPC ensembles to achieve 100-1000x faster biomolecular simulations](https://mlsanity.com/paper/2104.04797/) (2021)
- [Enabling Dynamic and Intelligent Workflows for HPC, Data Analytics, and AI Convergence](https://mlsanity.com/paper/2204.09287/) (2022)
- [Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computation](https://mlsanity.com/paper/1902.10810/) (2019)
- [Object Proxy Patterns for Accelerating Distributed Applications](https://mlsanity.com/paper/2407.01764/) (2024)
- [RADICAL-Pilot and Parsl: Executing Heterogeneous Workflows on HPC Platforms](https://mlsanity.com/paper/2105.13185/) (2021)
- [Accelerating Communications in Federated Applications with Transparent Object Proxies](https://mlsanity.com/paper/2305.09593/) (2023)
- [libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculations](https://mlsanity.com/paper/2104.08322/) (2021)
- [Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources](https://mlsanity.com/paper/2303.08803/) (2023)
- [Decision-Making Under Uncertainty for Multi-stage Pipelines: Simulation Studies to Benchmark Screening Strategies](https://mlsanity.com/paper/2203.01143/) (2022)

## Built on by (most cited first, 3 of 3)

- [Fantastic Scientific Agents and How to Build Them: AgentBuild for Rietveld Refinement](https://mlsanity.com/paper/2606.12834/) (2026)
- [Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applications](https://mlsanity.com/paper/2503.13343/) (2025)
- [Multi-Agent Discovery and Resource-Aware Autonomous Exploration of Scientific Datasets](https://mlsanity.com/paper/2608.22045/) (2026)

## Abstract (arXiv)

Computational workflows are a common class of application on supercomputers, yet the loosely coupled and heterogeneous nature of workflows often fails to take full advantage of their capabilities. We created Colmena to leverage the massive parallelism of a supercomputer by using Artificial Intelligence (AI) to learn from and adapt a workflow as it executes. Colmena allows scientists to define how their application should respond to events (e.g., task completion) as a series of cooperative agents. In this paper, we describe the design of Colmena, the challenges we overcame while deploying applications on exascale systems, and the science workflows we have enhanced through interweaving AI. The scaling challenges we discuss include developing steering strategies that maximize node utilization, introducing data fabrics that reduce communication overhead of data-intensive tasks, and implementing workflow tasks that cache costly operations between invocations. These innovations coupled with a variety of application patterns accessible through our agent-based steering model have enabled science advances in chemistry, biophysics, and materials science using different types of AI. Our vision is that Colmena will spur creative solutions that harness AI across many domains of scientific computing.