# A Comprehensive Perspective on Pilot-Job Systems

arXiv: https://arxiv.org/abs/1508.04180 · HTML: https://mlsanity.com/paper/1508.04180/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2015.

## Built on by (most cited first, 3 of 3)

- [Coupling streaming AI and HPC ensembles to achieve 100-1000x faster biomolecular simulations](https://mlsanity.com/paper/2104.04797/) (2021)
- [BFTrainer: Low-Cost Training of Neural Networks on Unfillable Supercomputer Nodes](https://mlsanity.com/paper/2106.12091/) (2021)
- [Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applications](https://mlsanity.com/paper/2503.13343/) (2025)

## Abstract (arXiv)

Pilot-Job systems play an important role in supporting distributed scientific computing. They are used to consume more than 700 million CPU hours a year by the Open Science Grid communities, and by processing up to 1 million jobs a day for the ATLAS experiment on the Worldwide LHC Computing Grid. With the increasing importance of task-level parallelism in high-performance computing, Pilot-Job systems are also witnessing an adoption beyond traditional domains. Notwithstanding the growing impact on scientific research, there is no agreement upon a definition of Pilot-Job system and no clear understanding of the underlying abstraction and paradigm. Pilot-Job implementations have proliferated with no shared best practices or open interfaces and little interoperability. Ultimately, this is hindering the realization of the full impact of Pilot-Jobs by limiting their robustness, portability, and maintainability. This paper offers a comprehensive analysis of Pilot-Job systems critically assessing their motivations, evolution, properties, and implementation. The three main contributions of this paper are: (i) an analysis of the motivations and evolution of Pilot-Job systems; (ii) an outline of the Pilot abstraction, its distinguishing logical components and functionalities, its terminology, and its architecture pattern; and (iii) the description of core and auxiliary properties of Pilot-Jobs systems and the analysis of seven exemplar Pilot-Job implementations. Together, these contributions illustrate the Pilot paradigm, its generality, and how it helps to address some challenges in distributed scientific computing.