# Resource Management for Low-latency Cooperative Fine-tuning of Foundation Models at the Network Edge

arXiv: https://arxiv.org/abs/2407.09873 · HTML: https://mlsanity.com/paper/2407.09873/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 13 Jul 2024.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Client-Edge-Cloud Hierarchical Federated Learning](https://mlsanity.com/paper/1905.06641/) (2019)
- [Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.15647/) (2023)
- [Asynchronous Stochastic Gradient Descent with Delay Compensation](https://mlsanity.com/paper/1609.08326/) (2016)
- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [Communication-Efficient Edge AI: Algorithms and Systems](https://mlsanity.com/paper/2002.09668/) (2020)
- [Towards an Intelligent Edge: Wireless Communication Meets Machine Learning](https://mlsanity.com/paper/1809.00343/) (2018)
- [Memory-Efficient Backpropagation Through Time](https://mlsanity.com/paper/1606.03401/) (2016)
- [Split Learning over Wireless Networks: Parallel Design and Resource Management](https://mlsanity.com/paper/2204.08119/) (2022)
- [Split Learning in 6G Edge Networks](https://mlsanity.com/paper/2306.12194/) (2023)
- [Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2404.06448/) (2024)
- [Wireless Communications for Collaborative Federated Learning](https://mlsanity.com/paper/2006.02499/) (2020)
- [When Large Language Model Agents Meet 6G Networks: Perception, Grounding, and Alignment](https://mlsanity.com/paper/2401.07764/) (2024)
- [Task-Oriented Sensing, Computation, and Communication Integration for Multi-Device Edge AI](https://mlsanity.com/paper/2207.00969/) (2022)
- [Big AI Models for 6G Wireless Networks: Opportunities, Challenges, and Research Directions](https://mlsanity.com/paper/2308.06250/) (2023)
- [Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks](https://mlsanity.com/paper/2407.02924/) (2024)
- [Mobile Edge Intelligence and Computing for the Internet of Vehicles](https://mlsanity.com/paper/1906.00400/) (2019)
- [Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning](https://mlsanity.com/paper/2003.04544/) (2020)
- [Computation Rate Maximization for Wireless Powered Mobile-Edge Computing with Binary Computation Offloading](https://mlsanity.com/paper/1708.08810/) (2017)
- [Energy-Efficient Processing and Robust Wireless Cooperative Transmission for Edge Inference](https://mlsanity.com/paper/1907.12475/) (2019)
- [Device-Edge Cooperative Fine-Tuning of Foundation Models as a 6G Service](https://mlsanity.com/paper/2310.18602/) (2023)

## Built on by (most cited first, 4 of 4)

- [FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuning](https://mlsanity.com/paper/2510.12078/) (2025)
- [Deploying Large AI Models on Resource-Limited Devices with Split Federated Learning](https://mlsanity.com/paper/2504.09114/) (2025)
- [Fine-Grained AI Model Caching and Downloading With Coordinated Multipoint Broadcasting in Multi-Cell Edge Networks](https://mlsanity.com/paper/2509.19341/) (2025)
- [Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge](https://mlsanity.com/paper/2606.04581/) (2026)

## Abstract (arXiv)

The emergence of large-scale foundation models (FoMo's) that can perform human-like intelligence motivates their deployment at the network edge for devices to access state-of-the-art artificial intelligence. For better user experiences, the pre-trained FoMo's need to be adapted to specialized downstream tasks through fine-tuning techniques. To transcend a single device's memory and computation limitations, we advocate multi-device cooperation within the device-edge cooperative fine-tuning (DEFT) paradigm, where edge devices cooperate to simultaneously optimize different parts of fine-tuning parameters within a FoMo. However, the parameter blocks reside at different depths within a FoMo architecture, leading to varied computation latency-and-memory cost due to gradient backpropagation-based calculations. The heterogeneous on-device computation and memory capacities and channel conditions necessitate an integrated communication-and-computation allocation of local computation loads and communication resources to achieve low-latency (LoLa) DEFT. To this end, we consider the depth-ware DEFT block allocation problem. The involved optimal block-device matching is tackled by the proposed low-complexity Cutting-RecoUNting-CHecking (CRUNCH) algorithm, which is designed by exploiting the monotone-increasing property between block depth and computation latency-and-memory cost. Next, the joint bandwidth-and-block allocation makes the problem more sophisticated. We observe a splittable Lagrangian expression through the transformation and analysis of the original problem, where the variables indicating device involvement are introduced. Then, the dual ascent method is employed to tackle this problem iteratively. Through extensive experiments conducted on the GLUE benchmark, our results demonstrate significant latency reduction achievable by LoLa DEFT for fine-tuning a RoBERTa model.