# Matching DNN Compression and Cooperative Training with Resources and Data Availability

arXiv: https://arxiv.org/abs/2212.02304 · HTML: https://mlsanity.com/paper/2212.02304/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 28 of 28)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [Towards moderate overparameterization: global convergence guarantees for training shallow neural networks](https://mlsanity.com/paper/1902.04674/) (2019)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [An LSTM Network for Highway Trajectory Prediction](https://mlsanity.com/paper/1801.07962/) (2018)
- [Fast-Convergent Federated Learning with Adaptive Weighting](https://mlsanity.com/paper/2012.00661/) (2020)
- [Personalized Federated Learning through Local Memorization](https://mlsanity.com/paper/2111.09360/) (2021)
- [Beyond expectation: Deep joint mean and quantile regression for spatio-temporal problems](https://mlsanity.com/paper/1808.08798/) (2018)
- [Communication-Efficient Federated Learning with Compensated Overlap-FedAvg](https://mlsanity.com/paper/2012.06706/) (2020)
- [RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Merging](https://mlsanity.com/paper/2110.01397/) (2021)
- [FedAR: Activity and Resource-Aware Federated Learning Model for Distributed Mobile Robots](https://mlsanity.com/paper/2101.03705/) (2021)
- [Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation](https://mlsanity.com/paper/2109.00525/) (2021)
- [A Privacy-Preserving-Oriented DNN Pruning and Mobile Acceleration Framework](https://mlsanity.com/paper/2003.06513/) (2020)
- [Federated Learning at the Network Edge: When Not All Nodes are Created Equal](https://mlsanity.com/paper/2101.01995/) (2021)
- [AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learning](https://mlsanity.com/paper/2112.01637/) (2021)
- [Energy-aware Resource Management for Federated Learning in Multi-access Edge Computing Systems](https://mlsanity.com/paper/2103.13293/) (2021)
- [KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2103.14891/) (2021)
- [Federated Deep Reinforcement Learning for the Distributed Control of NextG Wireless Networks](https://mlsanity.com/paper/2112.03465/) (2021)
- [Dynamic Network-Assisted D2D-Aided Coded Distributed Learning](https://mlsanity.com/paper/2111.14789/) (2021)
- [SmartDet: Context-Aware Dynamic Control of Edge Task Offloading for Mobile Object Detection](https://mlsanity.com/paper/2201.04235/) (2022)

## Built on by (most cited first, 4 of 4)

- [First-Order Dynamic Optimization for Streaming Convex Costs](https://mlsanity.com/paper/2310.07925/) (2023)
- [Edge-device Collaborative Computing for Multi-view Classification](https://mlsanity.com/paper/2409.15973/) (2024)
- [Dependable Distributed Training of Compressed Machine Learning Models](https://mlsanity.com/paper/2402.14346/) (2024)
- [Combining Relevance and Magnitude for Resource-Aware DNN Pruning](https://mlsanity.com/paper/2405.13088/) (2024)

## Abstract (arXiv)

To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and em where its training should be executed, are hard decisions to make, as they depend on the model itself, the resources of the available nodes, and the data such nodes own. Existing studies focus on each of those aspects individually, however, they do not account for how such decisions can be made jointly and adapted to one another. In this work, we model the network system focusing on the training of DNNs, formalize the above multi-dimensional problem, and, given its NP-hardness, formulate an approximate dynamic programming problem that we solve through the PACT algorithmic framework. Importantly, PACT leverages a time-expanded graph representing the learning process, and a data-driven and theoretical approach for the prediction of the loss evolution to be expected as a consequence of training decisions. We prove that PACT's solutions can get as close to the optimum as desired, at the cost of an increased time complexity, and that, in any case, such complexity is polynomial. Numerical results also show that, even under the most disadvantageous settings, PACT outperforms state-of-the-art alternatives and closely matches the optimal energy cost.