# Toward Real-time Analysis of Experimental Science Workloads on Geographically Distributed Supercomputers

arXiv: https://arxiv.org/abs/2105.06571 · HTML: https://mlsanity.com/paper/2105.06571/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2021.

## Built on by (most cited first, 1 of 1)

- [Bridging Data Center AI Systems with Edge Computing for Actionable Information Retrieval](https://mlsanity.com/paper/2105.13967/) (2021)

## Abstract (arXiv)

Massive upgrades to science infrastructure are driving data velocities upwards while stimulating adoption of increasingly data-intensive analytics. While next-generation exascale supercomputers promise strong support for I/O-intensive workflows, HPC remains largely untapped by live experiments, because data transfers and disparate batch-queueing policies are prohibitive when faced with scarce instrument time. To bridge this divide, we introduce Balsam: a distributed orchestration platform enabling workflows at the edge to securely and efficiently trigger analytics tasks across a user-managed federation of HPC execution sites. We describe the architecture of the Balsam service, which provides a workflow management API, and distributed sites that provision resources and schedule scalable, fault-tolerant execution. We demonstrate Balsam in efficiently scaling real-time analytics from two DOE light sources simultaneously onto three supercomputers (Theta, Summit, and Cori), while maintaining low overheads for on-demand computing, and providing a Python library for seamless integration with existing ecosystems of data analysis tools.