CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks
Not on the map. Why?
Abstract (arXiv)
The increasing demand for intelligent mobile applications has made multi-agent collaboration with Transformer-based large language models (LLMs) essential in mobile edge computing (MEC) networks. However, training LLMs in such environments remains challenging due to heavy computation, high end-to-end latency, and limited model generalization. We introduce CollaPipe, a hybrid distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving intelligent networks. In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks. Then we perform global model update via federated aggregation. To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power. We derive and use a closed-form convergence bound to design an Dynamic Segment Scheduling and Resource Allocation (DSSDA) algorithm based on Lyapunov optimization, ensuring system stability under long-term constraints. Extensive experiments on downstream tasks with Transformer and BERT models show that CollaPipe improves computation efficiency by up to 15.09%, reduces end-to-end latency by at least 48.98%, and cuts single device memory usage by more than half, enabling online learning in heterogeneous and dynamic communication environments.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 62 references that are in the map. First posted on arXiv 24 Sep 2025.
Builds on
28 of 62 references.
References · 28
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 263
- 2019Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocationcited by 94
- 2020Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learningcited by 90
- 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
- 2020Convergence of Update Aware Device Scheduling for Federated Learning at the Wireless Edgecited by 71
- 2023Split Learning in 6G Edge Networkscited by 68
- 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 50
- 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
16 more
- 2013Petuum: A New Platform for Distributed Machine Learning on Big Datacited by 35
- 2024Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networkscited by 25
- 2020Efficient Algorithms for Device Placement of DNN Graph Operatorscited by 24
- 2020Is Network the Bottleneck of Distributed Training?cited by 20
- 2022On Optimizing the Communication of Model Parallelismcited by 19
- 2022Low-latency Federated Learning with DNN Partition in Distributed Industrial IoT Networkscited by 14
- 2025Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Surveycited by 14
- 2024When IoT Meet LLMs: Applications and Challengescited by 11
- 2023Energy Efficient Semantic Communication over Wireless Networks with Rate Splittingcited by 9
- 2024Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networkscited by 7
- 2014Parallel Successive Convex Approximation for Nonsmooth Nonconvex Optimizationcited by 6
- 2024R-SFLLM: Jamming Resilient Framework for Split Federated Learning with Large Language Modelscited by 5
- 2025Optimizing Communication and Device Clustering for Clustered Federated Learning with Differential Privacycited by 4
- 2023Federated Learning Assisted Deep Q-Learning for Joint Task Offloading and Fronthaul Segment Routing in Open RANcited by 3
- 2024CityGPT: Towards Urban IoT Learning, Analysis and Interaction with Multi-Agent Systemcited by 3
- 2025A New Pathway to Integrated Learning and Communication (ILAC): Large AI Model and Hyperdimensional Computing for Communicationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.