mlsanityData to 13 Oct 2026

DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems

arXiv 2112.08761 ↗First posted 16 Dec 2021cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We study the problem of distributed training of neural networks (NNs) on devices with heterogeneous, limited, and time-varying availability of computational resources. We present an adaptive, resource-aware, on-device learning mechanism, DISTREAL, which is able to fully and efficiently utilize the available resources on devices in a distributed manner, increasing the convergence speed. This is achieved with a dropout mechanism that dynamically adjusts the computational complexity of training an NN by randomly dropping filters of convolutional layers of the model. Our main contribution is the introduction of a design space exploration (DSE) technique, which finds Pareto-optimal per-layer dropout vectors with respect to resource requirements and convergence speed of the training. Applying this technique, each device is able to dynamically select the dropout vector that fits its available resource without requiring any assistance from the server. We implement our solution in a federated learning (FL) system, where the availability of computational resources varies both between devices and over time, and show through extensive evaluation that we are able to significantly increase the convergence speed over the state of the art without compromising on the final accuracy.

Key facts. Cited by 10 papers in the map (data through 13 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

35 of 59 references.

References · 35

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  3. 2016Densely Connected Convolutional Networkscited by 6,707
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,426
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,836
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,981
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  8. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  9. 2016Deep Networks with Stochastic Depthcited by 977
  10. 2019Towards Federated Learning at Scale: System Designcited by 830
  11. 2018LEAF: A Benchmark for Federated Settingscited by 789
  12. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
23 more
  1. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  2. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  3. 2018Slimmable Neural Networkscited by 302
  4. 2019Asynchronous Federated Optimizationcited by 265
  5. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 226
  6. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 204
  7. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  8. 2019FedHealth: A Federated Transfer Learning Framework for Wearable Healthcarecited by 182
  9. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  10. 2018Deep Learning in Mobile and Wireless Networking: A Surveycited by 118
  11. 2020Communication-Efficient Edge AI: Algorithms and Systemscited by 95
  12. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  13. 2021HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmarkcited by 83
  14. 2021Dynamic Slimmable Networkcited by 83
  15. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 77
  16. 2019On-Device Machine Learning: An Algorithms and Learning Theory Perspectivecited by 63
  17. 2016Runtime Configurable Deep Neural Networks for Energy-Accuracy Trade-offcited by 19
  18. 2020Federated Learning for Resource-Constrained IoT Devices: Panoramas and State-of-the-artcited by 19
  19. 2019Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaborationcited by 15
  20. 2019Compressing RNNs for IoT devices by 15-38x using Kronecker Productscited by 11
  21. 2015Efficient batchwise dropout training using submatricescited by 3
  22. 2018Approximate Random Dropoutcited by 3
  23. 2020Distributed Learning on Heterogeneous Resource-Constrained Devicescited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 1; 2025: 3.

10 papers.

  1. 2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
  2. 2023FLuID: Mitigating Stragglers in Federated Learning using Invariant Dropoutcited by 16
  3. 2023FedZero: Leveraging Renewable Excess Energy in Federated Learningcited by 11
  4. 2022CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantizationcited by 7
  5. 2023Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devicescited by 7
  6. 2022Federated Learning for Inference at Anytime and Anywherecited by 6
  7. 2024FedTrans: Efficient Federated Learning via Multi-Model Transformationcited by 1
  8. 2025Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learningcited by 1
  9. 2025THOR: A Generic Energy Estimation Approach for On-Device Trainingcited by 0
  10. 2025Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Modelscited by 0