# PRISM: Probabilistic Runtime Insights and Scalable Performance Modeling for Large-Scale Distributed Training

arXiv: https://arxiv.org/abs/2510.15596 · HTML: https://mlsanity.com/paper/2510.15596/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2025.

## Built on by (most cited first, 1 of 1)

- [ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training](https://mlsanity.com/paper/2605.24326/) (2026)

## Abstract (arXiv)

Large model training beyond tens of thousands of GPUs is an uncharted territory. At such scales, disruptions to the training process are not a matter of if, but a matter of when -- a stochastic process degrading training productivity. Dynamic runtime variation will become increasingly more frequent as training scales and GPUs are operated in increasingly power-limited and thermally-stressed environments. At the 64,000+ GPU scale, we already observe 12% variability for frontier foundation model training. Motivated by our analysis and the large design space around performance variability, we present PRISM -- a performance modeling framework that captures the stochastic nature of large-scale distributed training. The core of PRISM is a statistical model that composes operator-level latency distributions through workload dependencies. Across 14 diverse training configurations spanning hundreds to 64K+ GPUs, PRISM estimates p95 execution time within 5.4% error. Using PRISM, we explore the design and optimization space of distributed training, enabling principled, variability-aware recommendations that can improve performance and system efficiency at scale.