# Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms

arXiv: https://arxiv.org/abs/2103.00091 · HTML: https://mlsanity.com/paper/2103.00091/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2021.

## Built on by (most cited first, 8 of 8)

- [Coupling streaming AI and HPC ensembles to achieve 100-1000x faster biomolecular simulations](https://mlsanity.com/paper/2104.04797/) (2021)
- [Pandemic Drugs at Pandemic Speed: Infrastructure for Accelerating COVID-19 Drug Discovery with Hybrid Machine Learning- and Physics-based Simulations on High Performance Computers](https://mlsanity.com/paper/2103.02843/) (2021)
- [Flight: A FaaS-Based Framework for Complex and Hierarchical Federated Learning](https://mlsanity.com/paper/2409.16495/) (2024)
- [AAFLOW: Scalable Patterns for Agentic AI Workflows](https://mlsanity.com/paper/2605.02162/) (2026)
- [[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows](https://mlsanity.com/paper/2607.10987/) (2026)
- [OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows](https://mlsanity.com/paper/2608.08340/) (2026)
- [Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applications](https://mlsanity.com/paper/2503.13343/) (2025)
- [Adaptive Protein Design Protocols and Middleware](https://mlsanity.com/paper/2510.06396/) (2025)

## Abstract (arXiv)

Many extreme scale scientific applications have workloads comprised of a large number of individual high-performance tasks. The Pilot abstraction decouples workload specification, resource management, and task execution via job placeholders and late-binding. As such, suitable implementations of the Pilot abstraction can support the collective execution of large number of tasks on supercomputers. We introduce RADICAL-Pilot (RP) as a portable, modular and extensible pilot-enabled runtime system. We describe RP's design, architecture and implementation. We characterize its performance and show its ability to scalably execute workloads comprised of tens of thousands heterogeneous tasks on DOE and NSF leadership-class HPC platforms. Specifically, we investigate RP's weak/strong scaling with CPU/GPU, single/multi core, (non)MPI tasks and Python functions when using most of ORNL Summit and TACC Frontera. RADICAL-Pilot can be used stand-alone, as well as the runtime for third-party workflow systems.