# ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training

arXiv: https://arxiv.org/abs/2510.00606 · HTML: https://mlsanity.com/paper/2510.00606/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2025.

## Built on by (most cited first, 2 of 2)

- [Unifying Local Communications and Local Updates for LLM Pretraining](https://mlsanity.com/paper/2606.11081/) (2026)
- [Belayer: Efficient Fault Tolerance for LLM Agentic RL Training](https://mlsanity.com/paper/2608.14635/) (2026)

## Abstract (arXiv)

Large-scale LLM pretraining now runs across 105--106 accelerators, making failures routine and elasticity mandatory. We posit that an elastic-native training system must jointly deliver (i) parameter consistency, (ii) low mean time to recovery (MTTR), (iii) high post-change throughput, and (iv) computation consistency. No prior system achieves all four simultaneously. To achieve these goals, we present ElasWave, which delivers per-step fault tolerance via multi-dimensional scheduling across graph, dataflow, DVFS, and RNG. ElasWave reshapes and reshards micro-batches while preserving the global batch size and gradient scale. It performs online pipeline resharding with asynchronous parameter migration and interleaves ZeRO partitions, reducing parameter recovery processes to disjoint rank-to-rank transfers. It further leverages DVFS to absorb pipeline bubbles and reshards RNG to keep computation consistency. Together, a dynamic communicator enables in-place communication group edits, while per-step in-memory snapshots support online verification and redistribution. We evaluate ElasWave on 96 NPUs and benchmark it against state-of-the-art baselines: throughput improves by 1.35× over ReCycle and 1.60× over TorchFT; communicator recovery completes within one second (up to 82×/3.6× faster than full/partial rebuilds); migration MTTR drops by as much as 51%; and convergence deviation is reduced by approximately 78%.