# Dependable Distributed Training of Compressed Machine Learning Models

arXiv: https://arxiv.org/abs/2402.14346 · HTML: https://mlsanity.com/paper/2402.14346/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Federated Optimization:Distributed Optimization Beyond the Datacenter](https://mlsanity.com/paper/1511.03575/) (2015)
- [Fast-Convergent Federated Learning with Adaptive Weighting](https://mlsanity.com/paper/2012.00661/) (2020)
- [Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy](https://mlsanity.com/paper/2103.03014/) (2021)
- [Personalized Federated Learning through Local Memorization](https://mlsanity.com/paper/2111.09360/) (2021)
- [Robust Federated Learning with Noisy Communication](https://mlsanity.com/paper/1911.00251/) (2019)
- [Communication-Efficient Federated Learning with Compensated Overlap-FedAvg](https://mlsanity.com/paper/2012.06706/) (2020)
- [DropNet: Reducing Neural Network Complexity via Iterative Pruning](https://mlsanity.com/paper/2207.06646/) (2022)
- [Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Control](https://mlsanity.com/paper/2310.10299/) (2023)
- [FedAR: Activity and Resource-Aware Federated Learning Model for Distributed Mobile Robots](https://mlsanity.com/paper/2101.03705/) (2021)
- [Impact of Training Dataset Size on Neural Answer Selection Models](https://mlsanity.com/paper/1901.10496/) (2019)
- [Matching DNN Compression and Cooperative Training with Resources and Data Availability](https://mlsanity.com/paper/2212.02304/) (2022)
- [Approximation Algorithms for the NFV Service Distribution Problem](https://mlsanity.com/paper/2105.10061/) (2021)
- [Efficient Distributed DNNs in the Mobile-edge-cloud Continuum](https://mlsanity.com/paper/2202.11349/) (2022)

## Abstract (arXiv)

The existing work on the distributed training of machine learning (ML) models has consistently overlooked the distribution of the achieved learning quality, focusing instead on its average value. This leads to a poor dependability}of the resulting ML models, whose performance may be much worse than expected. We fill this gap by proposing DepL, a framework for dependable learning orchestration, able to make high-quality, efficient decisions on (i) the data to leverage for learning, (ii) the models to use and when to switch among them, and (iii) the clusters of nodes, and the resources thereof, to exploit. For concreteness, we consider as possible available models a full DNN and its compressed versions. Unlike previous studies, DepL guarantees that a target learning quality is reached with a target probability, while keeping the training cost at a minimum. We prove that DepL has constant competitive ratio and polynomial complexity, and show that it outperforms the state-of-the-art by over 27% and closely matches the optimum.