# Rethinking Resource Management in Edge Learning: A Joint Pre-training and Fine-tuning Design Paradigm

arXiv: https://arxiv.org/abs/2404.00836 · HTML: https://mlsanity.com/paper/2404.00836/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [Statistical Aspects of Wasserstein Distances](https://mlsanity.com/paper/1806.05500/) (2018)
- [On the Theory of Transfer Learning: The Importance of Task Diversity](https://mlsanity.com/paper/2006.11650/) (2020)
- [HFEL: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learning](https://mlsanity.com/paper/2002.11343/) (2020)
- [Large Language Models Empowered Autonomous Edge AI for Connected Intelligence](https://mlsanity.com/paper/2307.02779/) (2023)
- [Large Generative AI Models for Telecom: The Next Big Thing?](https://mlsanity.com/paper/2306.10249/) (2023)
- [To Talk or to Work: Flexible Communication Compression for Energy Efficient Federated Learning over Heterogeneous Mobile Edge Devices](https://mlsanity.com/paper/2012.11804/) (2020)
- [Energy-Efficient Resource Management for Federated Edge Learning with CPU-GPU Heterogeneous Computing](https://mlsanity.com/paper/2007.07122/) (2020)
- [Scaling Federated Learning for Fine-tuning of Large Language Models](https://mlsanity.com/paper/2102.00875/) (2021)
- [Dynamic Scheduling for Over-the-Air Federated Edge Learning with Energy Constraints](https://mlsanity.com/paper/2106.00490/) (2021)
- [Energy-Efficient Federated Edge Learning with Joint Communication and Computation Design](https://mlsanity.com/paper/2003.00199/) (2020)
- [Data-Importance Aware User Scheduling for Communication-Efficient Edge Machine Learning](https://mlsanity.com/paper/1910.02214/) (2019)
- [Gradient and Channel Aware Dynamic Scheduling for Over-the-Air Computation in Federated Edge Learning Systems](https://mlsanity.com/paper/2212.00491/) (2022)
- [Low-latency Federated Learning with DNN Partition in Distributed Industrial IoT Networks](https://mlsanity.com/paper/2210.14700/) (2022)
- [Knowledge-aided Federated Learning for Energy-limited Wireless Networks](https://mlsanity.com/paper/2209.12277/) (2022)
- [Vertical Federated Learning over Cloud-RAN: Convergence Analysis and System Optimization](https://mlsanity.com/paper/2305.06279/) (2023)
- [Analysis and Optimization of Wireless Federated Learning with Data Heterogeneity](https://mlsanity.com/paper/2308.03521/) (2023)
- [On the Energy and Communication Efficiency Tradeoffs in Federated and Multi-Task Learning](https://mlsanity.com/paper/2212.01049/) (2022)
- [GQFedWAvg: Optimization-Based Quantized Federated Learning in General Edge Computing Systems](https://mlsanity.com/paper/2306.07497/) (2023)

## Built on by (most cited first, 13 of 13)

- [Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2404.06448/) (2024)
- [PianoBART: Symbolic Piano Music Generation and Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2407.03361/) (2024)
- [CSI-BERT2: A BERT-inspired Framework for Efficient CSI Prediction and Classification in Wireless Communication and Sensing](https://mlsanity.com/paper/2412.06861/) (2024)
- [The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networks](https://mlsanity.com/paper/2505.09214/) (2025)
- [Empowering Intelligent Low-altitude Economy with Large AI Model Deployment](https://mlsanity.com/paper/2505.22343/) (2025)
- [Deep Learning-based Techniques for Integrated Sensing and Communication Systems: State-of-the-Art, Challenges, and Opportunities](https://mlsanity.com/paper/2509.06968/) (2025)
- [Quantization-Aware Collaborative Inference for Large Embodied AI Models](https://mlsanity.com/paper/2602.13052/) (2026)
- [Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)](https://mlsanity.com/paper/2409.16590/) (2024)
- [Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)](https://mlsanity.com/paper/2409.16594/) (2024)
- [Fine-Grained AI Model Caching and Downloading With Coordinated Multipoint Broadcasting in Multi-Cell Edge Networks](https://mlsanity.com/paper/2509.19341/) (2025)
- [EvoMail: Self-Evolving Cognitive Agents for Adaptive Spam and Phishing Email Defense](https://mlsanity.com/paper/2509.21129/) (2025)
- [Closing the Generalization Gap in Parameter-efficient Federated Edge Learning](https://mlsanity.com/paper/2511.23282/) (2025)
- [Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge](https://mlsanity.com/paper/2606.04581/) (2026)

## Abstract (arXiv)

In some applications, edge learning is experiencing a shift in focusing from conventional learning from scratch to new two-stage learning unifying pre-training and task-specific fine-tuning. This paper considers the problem of joint communication and computation resource management in a two-stage edge learning system. In this system, model pre-training is first conducted at an edge server via centralized learning on local pre-stored general data, and then task-specific fine-tuning is performed at edge devices based on the pre-trained model via federated edge learning. For the two-stage learning model, we first analyze the convergence behavior (in terms of the average squared gradient norm bound), which characterizes the impacts of various system parameters such as the number of learning rounds and batch sizes in the two stages on the convergence rate. Based on our analytical results, we then propose a joint communication and computation resource management design to minimize an average squared gradient norm bound, subject to constraints on the transmit power, overall system energy consumption, and training delay. The decision variables include the number of learning rounds, batch sizes, clock frequencies, and transmit power control for both pre-training and fine-tuning stages. Finally, numerical results are provided to evaluate the effectiveness of our proposed design. It is shown that the proposed joint resource management over the pre-training and fine-tuning stages well balances the system performance trade-off among the training accuracy, delay, and energy consumption. The proposed design is also shown to effectively leverage the inherent trade-off between pre-training and fine-tuning, which arises from the differences in data distribution between pre-stored general data versus real-time task-specific data, thus efficiently optimizing overall system performance.