# From Edge to HPC: Investigating Cross-Facility Data Streaming Architectures

arXiv: https://arxiv.org/abs/2509.24030 · HTML: https://mlsanity.com/paper/2509.24030/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 40 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Builds on (most cited first, 8 of 8)

- [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015)
- [Hyperparameter Search in Machine Learning](https://mlsanity.com/paper/1502.02127/) (2015)
- [Deep learning at the edge enables real-time streaming ptychographic imaging](https://mlsanity.com/paper/2209.09408/) (2022)
- [Coupling streaming AI and HPC ensembles to achieve 100-1000x faster biomolecular simulations](https://mlsanity.com/paper/2104.04797/) (2021)
- [AI-coupled HPC Workflow Applications, Middleware and Performance](https://mlsanity.com/paper/2406.14315/) (2024)
- [Secure API-Driven Research Automation to Accelerate Scientific Discovery](https://mlsanity.com/paper/2506.11950/) (2025)
- [EJ-FAT Joint ESnet JLab FPGA Accelerated Transport Load Balancer](https://mlsanity.com/paper/2303.16351/) (2023)
- [A Study on Messaging Trade-offs in Data Streaming for Scientific Workflows](https://mlsanity.com/paper/2509.07199/) (2025)

## Abstract (arXiv)

In this paper, we investigate three cross-facility data streaming architectures, Direct Streaming (DTS), Proxied Streaming (PRS), and Managed Service Streaming (MSS). We examine their architectural variations in data flow paths and deployment feasibility, and detail their implementation using the Data Streaming to HPC (DS2HPC) architectural framework and the SciStream memory-to-memory streaming toolkit on the production-grade Advanced Computing Ecosystem (ACE) infrastructure at Oak Ridge Leadership Computing Facility (OLCF). We present a workflow-specific evaluation of these architectures using three synthetic workloads derived from the streaming characteristics of scientific workflows. Through simulated experiments, we measure streaming throughput, round-trip time, and overhead under work sharing, work sharing with feedback, and broadcast and gather messaging patterns commonly found in AI-HPC communication motifs. Our study shows that DTS offers a minimal-hop path, resulting in higher throughput and lower latency, whereas MSS provides greater deployment feasibility and scalability across multiple users but incurs significant overhead. PRS lies in between, offering a scalable architecture whose performance matches DTS in most cases.