mlsanityData to 7 Oct 2026

OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters

arXiv 2503.17469 ↗First posted 21 Mar 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Deep learning systems are optimized for clusters with homogeneous resources. However, heterogeneity is prevalent in computing infrastructure across edge, cloud and HPC. When training neural networks using stochastic gradient descent techniques on heterogeneous resources, performance degrades due to stragglers and stale updates. In this work, we develop an adaptive batch-scaling framework called OmniLearn to mitigate the effects of heterogeneity in distributed training. Our approach is inspired by proportional controllers to balance computation across heterogeneous servers, and works under varying resource availability. By dynamically adjusting worker mini-batches at runtime, OmniLearn reduces training time by 14-85%. We also investigate asynchronous training, where our techniques improve accuracy by up to 6.9%.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

28 of 48 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  8. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  9. 2018An Empirical Model of Large-Batch Trainingcited by 268
  10. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  11. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  12. 2017Revisiting Distributed Synchronous SGDcited by 142
16 more
  1. 2014Deep learning with Elastic Averaging SGDcited by 90
  2. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 87
  3. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
  4. 2016Gossip training for deep learningcited by 33
  5. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  6. 2018Large batch size training of neural networks with adversarial training and second-order informationcited by 18
  7. 2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
  8. 2019Hop: Heterogeneity-Aware Decentralized Trainingcited by 10
  9. 2023GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Trainingcited by 10
  10. 2019Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operationscited by 9
  11. 2020Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimizationcited by 9
  12. 2020Characterizing and Modeling Distributed Training with Transient Cloud GPU Serverscited by 8
  13. 2023ScaDLES: Scalable Deep Learning over Streaming data at the Edgecited by 8
  14. 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
  15. 2023Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Trainingcited by 5
  16. 2023Flexible Communication for Optimal Distributed Learning over Unpredictable Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPCcited by 1
  2. 2025On Using Large-Batches in Federated Learningcited by 0
  3. 2026Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Trainingcited by 0
  4. 2026HiGFRL: Hierarchical Graph Fusion-Driven Reinforcement Learning for Dependency-Aware Task Scheduling in Heterogeneous Cloudcited by 0
  5. 2026HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Trainingcited by 0