PiPar: Pipeline Parallelism for Collaborative Machine Learning
Not on the map. Why?
Abstract (arXiv)
Collaborative machine learning (CML) techniques, such as federated learning, have been proposed to train deep learning models across multiple mobile devices and a server. CML techniques are privacy-preserving as a local model that is trained on each device instead of the raw data from the device is shared with the server. However, CML training is inefficient due to low resource utilization. We identify idling resources on the server and devices due to sequential computation and communication as the principal cause of low resource utilization. A novel framework PiPar that leverages pipeline parallelism for CML techniques is developed to substantially improve resource utilization. A new training pipeline is designed to parallelize the computations on different hardware resources and communication on different bandwidth resources, thereby accelerating the training process in CML. A low overhead automated parameter selection method is proposed to optimize the pipeline, maximizing the utilization of available resources. The experimental results confirm the validity of the underlying approach of PiPar and highlight that when compared to federated learning: (i) the idle time of the server can be reduced by up to 64.1x, and (ii) the overall training time can be accelerated by up to 34.6x under varying network conditions for a collection of six small and large popular deep neural networks and four datasets without sacrificing accuracy. It is also experimentally demonstrated that PiPar achieves performance benefits when incorporating differential privacy methods and operating in environments with heterogeneous devices and changing bandwidths.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 1 Dec 2022.
Builds on
30 of 47 references.
References · 30
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2017Mask R-CNNcited by 7,259
- 2016Deep Learning with Differential Privacycited by 2,682
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2019Searching for MobileNetV3cited by 1,984
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
18 more
- 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
- 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
- 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
- 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
- 2018Distributed learning of deep neural network over multiple agentscited by 277
- 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 193
- 2019Detailed comparison of communication efficiency of split learning and federated learningcited by 76
- 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
- 2020End-to-End Evaluation of Federated Learning and Split Learning for Internet of Thingscited by 64
- 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
- 2020Advancements of federated learning towards privacy preservation: from federated learning to split learningcited by 34
- 2020Communication-Efficient Federated Learning with Compensated Overlap-FedAvgcited by 31
- 2021FedAdapt: Adaptive Offloading for IoT Devices in Federated Learningcited by 26
- 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
- 2021Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learningcited by 22
- 2021Communication and Computation Reduction for Split Learning using Asynchronous Trainingcited by 11
- 2020Scission: Performance-driven and Context-aware Cloud-Edge Distribution of Deep Neural Networkscited by 1
Built on by
Citing papers per year: 2025: 2.
2 papers.