# ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training

arXiv: https://arxiv.org/abs/2605.24326 · HTML: https://mlsanity.com/paper/2605.24326/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 23 May 2026.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning](https://mlsanity.com/paper/2405.00451/) (2024)
- [Don't Use Large Mini-Batches, Use Local SGD](https://mlsanity.com/paper/1808.07217/) (2018)
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://mlsanity.com/paper/2311.08105/) (2023)
- [Revisiting Reliability in Large-Scale Machine Learning Research Clusters](https://mlsanity.com/paper/2410.21680/) (2024)
- [Asynchronous Local-SGD Training for Language Modeling](https://mlsanity.com/paper/2401.09135/) (2024)
- [Scaling Law for Language Models Training Considering Batch Size](https://mlsanity.com/paper/2412.01505/) (2024)
- [MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems](https://mlsanity.com/paper/2310.02784/) (2023)
- [Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters](https://mlsanity.com/paper/2504.17096/) (2025)
- [Power Stabilization for AI Training Datacenters](https://mlsanity.com/paper/2508.14318/) (2025)
- [FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression](https://mlsanity.com/paper/2410.12707/) (2024)
- [HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training](https://mlsanity.com/paper/2506.04531/) (2025)
- [I've Got 99 Problems But FLOPS Ain't One](https://mlsanity.com/paper/2407.12819/) (2024)
- [Improving training time and GPU utilization in geo-distributed language model training](https://mlsanity.com/paper/2411.14458/) (2024)
- [PAT: a new algorithm for all-gather and reduce-scatter operations at scale](https://mlsanity.com/paper/2506.20252/) (2025)
- [CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training](https://mlsanity.com/paper/2507.00217/) (2025)
- [Deconstructing the Tail at Scale Effect Across Network Protocols](https://mlsanity.com/paper/1701.03100/) (2017)
- [SDR-RDMA: Software-Defined Reliability Architecture for Planetary Scale RDMA Communication](https://mlsanity.com/paper/2505.05366/) (2025)
- [PRISM: Probabilistic Runtime Insights and Scalable Performance Modeling for Large-Scale Distributed Training](https://mlsanity.com/paper/2510.15596/) (2025)

## Abstract (arXiv)

The rapid scaling of large language model training requires distributing GPU resources across multiple data center buildings and regions. We refer to such paradigm as "scale-across" training. As infrastructure expands, the system design space becomes increasingly intricate, encompassing new model architectures, hardware heterogeneity, and evolving communication patterns. Drawing from Meta's production experience, we highlight the complexities of deploying training jobs across a few data centers housing hundreds of thousands of GPUs. To accelerate exploration of the large design space and to enable efficient training for frontier model development, we conduct in-depth characterization of three key design dimensions: parallelism placement, parallelism scheduling, and network layer technologies. We then propose ScaleAcross Explorer, an optimizer that considers the interplay of design dimensions and holistically optimizes scale-across training. Testbed experiments and simulations demonstrate up to 64.62% training speedups over production configuration and up to 37.59% training speedups over the state-of-the-art baseline across a wide range of design points.