# CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks

arXiv: https://arxiv.org/abs/2509.19855 · HTML: https://mlsanity.com/paper/2509.19855/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [A Joint Learning and Communications Framework for Federated Learning over Wireless Networks](https://mlsanity.com/paper/1909.07972/) (2019)
- [Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation](https://mlsanity.com/paper/1910.13067/) (2019)
- [Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learning](https://mlsanity.com/paper/2007.07174/) (2020)
- [FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model](https://mlsanity.com/paper/2406.17706/) (2024)
- [Convergence of Update Aware Device Scheduling for Federated Learning at the Wireless Edge](https://mlsanity.com/paper/2001.10402/) (2020)
- [Split Learning in 6G Edge Networks](https://mlsanity.com/paper/2306.12194/) (2023)
- [Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models](https://mlsanity.com/paper/2305.11414/) (2023)
- [HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism](https://mlsanity.com/paper/2005.14038/) (2020)
- [Petuum: A New Platform for Distributed Machine Learning on Big Data](https://mlsanity.com/paper/1312.7651/) (2013)
- [Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks](https://mlsanity.com/paper/2407.02924/) (2024)
- [Efficient Algorithms for Device Placement of DNN Graph Operators](https://mlsanity.com/paper/2006.16423/) (2020)
- [Is Network the Bottleneck of Distributed Training?](https://mlsanity.com/paper/2006.10103/) (2020)
- [On Optimizing the Communication of Model Parallelism](https://mlsanity.com/paper/2211.05322/) (2022)
- [Low-latency Federated Learning with DNN Partition in Distributed Industrial IoT Networks](https://mlsanity.com/paper/2210.14700/) (2022)
- [Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey](https://mlsanity.com/paper/2505.01821/) (2025)
- [When IoT Meet LLMs: Applications and Challenges](https://mlsanity.com/paper/2411.17722/) (2024)
- [Energy Efficient Semantic Communication over Wireless Networks with Rate Splitting](https://mlsanity.com/paper/2301.01987/) (2023)
- [Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networks](https://mlsanity.com/paper/2412.06414/) (2024)
- [Parallel Successive Convex Approximation for Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/1406.3665/) (2014)
- [R-SFLLM: Jamming Resilient Framework for Split Federated Learning with Large Language Models](https://mlsanity.com/paper/2407.11654/) (2024)
- [Optimizing Communication and Device Clustering for Clustered Federated Learning with Differential Privacy](https://mlsanity.com/paper/2507.07320/) (2025)
- [Federated Learning Assisted Deep Q-Learning for Joint Task Offloading and Fronthaul Segment Routing in Open RAN](https://mlsanity.com/paper/2308.07258/) (2023)
- [CityGPT: Towards Urban IoT Learning, Analysis and Interaction with Multi-Agent System](https://mlsanity.com/paper/2405.14691/) (2024)
- [A New Pathway to Integrated Learning and Communication (ILAC): Large AI Model and Hyperdimensional Computing for Communication](https://mlsanity.com/paper/2506.18432/) (2025)

## Built on by (most cited first, 1 of 1)

- [ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling](https://mlsanity.com/paper/2608.07974/) (2026)

## Abstract (arXiv)

The increasing demand for intelligent mobile applications has made multi-agent collaboration with Transformer-based large language models (LLMs) essential in mobile edge computing (MEC) networks. However, training LLMs in such environments remains challenging due to heavy computation, high end-to-end latency, and limited model generalization. We introduce CollaPipe, a hybrid distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving intelligent networks. In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks. Then we perform global model update via federated aggregation. To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power. We derive and use a closed-form convergence bound to design an Dynamic Segment Scheduling and Resource Allocation (DSSDA) algorithm based on Lyapunov optimization, ensuring system stability under long-term constraints. Extensive experiments on downstream tasks with Transformer and BERT models show that CollaPipe improves computation efficiency by up to 15.09%, reduces end-to-end latency by at least 48.98%, and cuts single device memory usage by more than half, enabling online learning in heterogeneous and dynamic communication environments.