# Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training

arXiv: https://arxiv.org/abs/2505.12815 · HTML: https://mlsanity.com/paper/2505.12815/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 15 of 15)

- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations](https://mlsanity.com/paper/2409.05976/) (2024)
- [Federated Learning Enables Big Data for Rare Cancer Boundary Detection](https://mlsanity.com/paper/2204.10836/) (2022)
- [When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods](https://mlsanity.com/paper/2212.10025/) (2022)
- [Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning](https://mlsanity.com/paper/2008.12260/) (2020)
- [Federated Learning for Breast Density Classification: A Real-World Implementation](https://mlsanity.com/paper/2009.01871/) (2020)
- [Data Heterogeneity-Robust Federated Learning via Group Client Selection in Industrial IoT](https://mlsanity.com/paper/2202.01512/) (2022)
- [Unicron: Economizing Self-Healing LLM Training at Scale](https://mlsanity.com/paper/2401.00134/) (2023)
- [DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Models](https://mlsanity.com/paper/2406.10707/) (2024)
- [DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters](https://mlsanity.com/paper/1909.06040/) (2019)
- [MimiC: Combating Client Dropouts in Federated Learning by Mimicking Central Updates](https://mlsanity.com/paper/2306.12212/) (2023)
- [Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelis](https://mlsanity.com/paper/2406.18820/) (2024)
- [Accelerating Geo-distributed Machine Learning with Network-Aware Adaptive Tree and Auxiliary Route](https://mlsanity.com/paper/2404.11352/) (2024)
- [Rubick: Exploiting Job Reconfigurability for Deep Learning Cluster Scheduling](https://mlsanity.com/paper/2408.08586/) (2024)
- [Communication-Efficient Federated Learning under Dynamic Device Arrival and Departure: Convergence Analysis and Algorithm Design](https://mlsanity.com/paper/2410.05662/) (2024)

## Abstract (arXiv)

Node and link churn in multi-party, cross-region clusters over wide-area networks (WANs) often disrupts distributed training. However, checkpoint-based recovery and cloud-centric autoscaling react slowly and assume centralized control, which is misaligned with the self-governed setup where institutions can freely join and leave. This paper proposes Chaos, a multi-party distributed training system with self-healing and autoscaling, enabling robust and elastic training under churn. It speeds up autoscaling via multi-neighbor state replication and model sharding. We formalize the sharding and assignment as a MINLP that captures WAN heterogeneity, and reduce it to a tractable MILP by analyzing its monotonicity on a divisibility chain. By establishing an equivalence, we derive a greedy algorithm that follows optimality rules and yields the optimal solution in polynomial time. Chaos uses a cluster monitor to track resource and topology changes, and handles scaling events through peer negotiation protocols, enabling fully self-governed autoscaling among institutions. Experiments show that Chaos has substantially lower scale-out delay than Pollux, Elan, and Autoscaling, and handles scale-in, connect-link, and disconnect-link events within 20ms. It also delivers the lowest idle time, showing superior resource use and scalability as the cluster grows.