# Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applications

arXiv: https://arxiv.org/abs/2503.13343 · HTML: https://mlsanity.com/paper/2503.13343/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Builds on (most cited first, 16 of 16)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [The Landscape and Challenges of HPC Research and LLMs](https://mlsanity.com/paper/2402.02018/) (2024)
- [Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms](https://mlsanity.com/paper/2103.00091/) (2021)
- [Enabling Dynamic and Intelligent Workflows for HPC, Data Analytics, and AI Convergence](https://mlsanity.com/paper/2204.09287/) (2022)
- [Learning Everywhere: A Taxonomy for the Integration of Machine Learning and Simulations](https://mlsanity.com/paper/1909.13340/) (2019)
- [AI-coupled HPC Workflow Applications, Middleware and Performance](https://mlsanity.com/paper/2406.14315/) (2024)
- [RADICAL-Pilot and Parsl: Executing Heterogeneous Workflows on HPC Platforms](https://mlsanity.com/paper/2105.13185/) (2021)
- [A Comprehensive Perspective on Pilot-Job Systems](https://mlsanity.com/paper/1508.04180/) (2015)
- [Employing Artificial Intelligence to Steer Exascale Workflows with Colmena](https://mlsanity.com/paper/2408.14434/) (2024)
- [Middleware Building Blocks for Workflow Systems](https://mlsanity.com/paper/1903.10057/) (2019)
- [Characterizing the Performance of Executing Many-tasks on Summit](https://mlsanity.com/paper/1909.03057/) (2019)
- [AI-coupled HPC Workflows](https://mlsanity.com/paper/2208.11745/) (2022)
- [Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis](https://mlsanity.com/paper/2501.15370/) (2025)
- [RAPTOR: Ravenous Throughput Computing](https://mlsanity.com/paper/2209.00114/) (2022)
- [Comparison of Autoscaling Frameworks for Containerised Machine-Learning-Applications in a Local and Cloud Environment](https://mlsanity.com/paper/2311.18659/) (2023)

## Abstract (arXiv)

Hybrid workflows combining traditional HPC and novel ML methodologies are transforming scientific computing. This paper presents the architecture and implementation of a scalable runtime system that extends RADICAL-Pilot with service-based execution to support AI-out-HPC workflows. Our runtime system enables distributed ML capabilities, efficient resource management, and seamless HPC/ML coupling across local and remote platforms. Preliminary experimental results show that our approach manages concurrent execution of ML models across local and remote HPC/cloud resources with minimal architectural overheads. This lays the foundation for prototyping three representative data-driven workflow applications and executing them at scale on leadership-class HPC platforms.