# Distributed Convolutional Neural Network Training on Mobile and Edge Clusters

arXiv: https://arxiv.org/abs/2409.09083 · HTML: https://mlsanity.com/paper/2409.09083/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Builds on (most cited first, 8 of 8)

- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning](https://mlsanity.com/paper/2007.11622/) (2020)
- [A Survey on Collaborative DNN Inference for Edge Intelligence](https://mlsanity.com/paper/2207.07812/) (2022)
- [All One Needs to Know about Fog Computing and Related Edge Computing Paradigms: A Complete Survey](https://mlsanity.com/paper/1808.05283/) (2018)
- [MobileTL: On-device Transfer Learning with Inverted Residual Blocks](https://mlsanity.com/paper/2212.03246/) (2022)
- [Distributed Training for Deep Learning Models On An Edge Computing Network Using ShieldedReinforcement Learning](https://mlsanity.com/paper/2206.00774/) (2022)
- [Fused Depthwise Tiling for Memory Optimization in TinyML Deep Neural Network Inference](https://mlsanity.com/paper/2303.17878/) (2023)

## Abstract (arXiv)

The training of deep and/or convolutional neural networks (DNNs/CNNs) is traditionally done on servers with powerful CPUs and GPUs. Recent efforts have emerged to localize machine learning tasks fully on the edge. This brings advantages in reduced latency and increased privacy, but necessitates working with resource-constrained devices. Approaches for inference and training in mobile and edge devices based on pruning, quantization or incremental and transfer learning require trading off accuracy. Several works have explored distributing inference operations on mobile and edge clusters instead. However, there is limited literature on distributed training on the edge. Existing approaches all require a central, potentially powerful edge or cloud server for coordination or offloading. In this paper, we describe an approach for distributed CNN training exclusively on mobile and edge devices. Our approach is beneficial for the initial CNN layers that are feature map dominated. It is based on partitioning forward inference and back-propagation operations among devices through tiling and fusing to maximize locality and expose communication and memory-aware parallelism. We also introduce the concept of layer grouping to further fine-tune performance based on computation and communication trade-off. Results show that for a cluster of 2-6 quad-core Raspberry Pi3 devices, training of an object-detection CNN provides a 2x-15x speedup with respect to a single core and up to 8x reduction in memory usage per device, all without sacrificing accuracy. Grouping offers up to 1.5x speedup depending on the reference profile and batch size.