mlsanityData to 14 Oct 2026

Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training

arXiv 2505.12815 ↗First posted 19 May 2025cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Node and link churn in multi-party, cross-region clusters over wide-area networks (WANs) often disrupts distributed training. However, checkpoint-based recovery and cloud-centric autoscaling react slowly and assume centralized control, which is misaligned with the self-governed setup where institutions can freely join and leave. This paper proposes Chaos, a multi-party distributed training system with self-healing and autoscaling, enabling robust and elastic training under churn. It speeds up autoscaling via multi-neighbor state replication and model sharding. We formalize the sharding and assignment as a MINLP that captures WAN heterogeneity, and reduce it to a tractable MILP by analyzing its monotonicity on a divisibility chain. By establishing an equivalence, we derive a greedy algorithm that follows optimality rules and yields the optimal solution in polynomial time. Chaos uses a cluster monitor to track resource and topology changes, and handles scaling events through peer negotiation protocols, enabling fully self-governed autoscaling among institutions. Experiments show that Chaos has substantially lower scale-out delay than Pollux, Elan, and Autoscaling, and handles scale-in, connect-link, and disconnect-link events within 20ms. It also delivers the lowest idle time, showing superior resource use and scalability as the cluster grows.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 37 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

15 of 37 references.

References · 15

  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  2. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  3. 2022Federated Learning Enables Big Data for Rare Cancer Boundary Detectioncited by 84
  4. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  5. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  6. 2020Federated Learning for Breast Density Classification: A Real-World Implementationcited by 62
  7. 2022Data Heterogeneity-Robust Federated Learning via Group Client Selection in Industrial IoTcited by 25
  8. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  9. 2024DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Modelscited by 16
  10. 2019DL2: A Deep Learning-driven Scheduler for Deep Learning Clusterscited by 14
  11. 2023MimiC: Combating Client Dropouts in Federated Learning by Mimicking Central Updatescited by 10
  12. 2024Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Paralleliscited by 9
3 more
  1. 2024Accelerating Geo-distributed Machine Learning with Network-Aware Adaptive Tree and Auxiliary Routecited by 5
  2. 2024Rubick: Exploiting Job Reconfigurability for Deep Learning Cluster Schedulingcited by 4
  3. 2024Communication-Efficient Federated Learning under Dynamic Device Arrival and Departure: Convergence Analysis and Algorithm Designcited by 1

Built on by

Not cited yet.