mlsanityData to 7 Oct 2026

ScaDLES: Scalable Deep Learning over Streaming data at the Edge

arXiv 2301.08897 ↗First posted 21 Jan 2023cs.DC cs.LG cs.NIcited by 8

Not on the map. Why?

Abstract (arXiv)

Distributed deep learning (DDL) training systems are designed for cloud and data-center environments that assumes homogeneous compute resources, high network bandwidth, sufficient memory and storage, as well as independent and identically distributed (IID) data across all nodes. However, these assumptions don't necessarily apply on the edge, especially when training neural networks on streaming data in an online manner. Computing on the edge suffers from both systems and statistical heterogeneity. Systems heterogeneity is attributed to differences in compute resources and bandwidth specific to each device, while statistical heterogeneity comes from unbalanced and skewed data on the edge. Different streaming-rates among devices can be another source of heterogeneity when dealing with streaming data. If the streaming rate is lower than training batch-size, device needs to wait until enough samples have streamed in before performing a single iteration of stochastic gradient descent (SGD). Thus, low-volume streams act like stragglers slowing down devices with high-volume streams in synchronous training. On the other hand, data can accumulate quickly in the buffer if the streaming rate is too high and the devices can't train at line-rate. In this paper, we introduce ScaDLES to efficiently train on streaming data at the edge in an online fashion, while also addressing the challenges of limited bandwidth and training with non-IID data. We empirically show that ScaDLES converges up to 3.29 times faster compared to conventional distributed SGD.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 21 Jan 2023.

Builds on

24 of 34 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2018Federated Learning with Non-IID Datacited by 1,099
  12. 2018LEAF: A Benchmark for Federated Settingscited by 780
12 more
  1. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  2. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  3. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  4. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  5. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  6. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  7. 2018The Convergence of Sparsified Gradient Methodscited by 228
  8. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  9. 2020The Early Phase of Neural Network Trainingcited by 114
  10. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  11. 2020Is Network the Bottleneck of Distributed Training?cited by 19
  12. 2020Accordion: Adaptive Gradient Communication via Critical Learning Regime Identificationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 4; 2026: 1.

8 papers.

  1. 2023GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Trainingcited by 10
  2. 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
  3. 2025OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusterscited by 5
  4. 2023Flexible Communication for Optimal Distributed Learning over Unpredictable Networkscited by 3
  5. 2025Onboard Optimization and Learning: A Surveycited by 1
  6. 2025OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPCcited by 1
  7. 2025On Using Large-Batches in Federated Learningcited by 0
  8. 2026Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Trainingcited by 0