# FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning

arXiv: https://arxiv.org/abs/2506.01001 · HTML: https://mlsanity.com/paper/2506.01001/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models](https://mlsanity.com/paper/2401.06432/) (2024)
- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods](https://mlsanity.com/paper/2212.10025/) (2022)
- [No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices](https://mlsanity.com/paper/2202.08036/) (2022)
- [Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization](https://mlsanity.com/paper/2403.12422/) (2024)
- [FedAdapter: Efficient Federated Learning for Modern NLP](https://mlsanity.com/paper/2205.10162/) (2022)
- [FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients](https://mlsanity.com/paper/2311.11227/) (2023)
- [Decentralized Low-Rank Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2501.15361/) (2025)
- [GoRA: Gradient-driven Adaptive Low Rank Adaptation](https://mlsanity.com/paper/2502.12171/) (2025)
- [DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation](https://mlsanity.com/paper/2402.17812/) (2024)
- [SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics](https://mlsanity.com/paper/2305.18513/) (2023)
- [Exploring Selective Layer Fine-Tuning in Federated Learning](https://mlsanity.com/paper/2408.15600/) (2024)

## Built on by (most cited first, 10 of 10)

- [Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning](https://mlsanity.com/paper/2506.05977/) (2025)
- [Verifiable Fine-Tuning for LLMs: Zero-Knowledge Training Proofs Bound to Data Provenance and Policy](https://mlsanity.com/paper/2510.16830/) (2025)
- [Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA](https://mlsanity.com/paper/2602.20492/) (2026)
- [Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning](https://mlsanity.com/paper/2604.26340/) (2026)
- [Socialized Learning and Emergent Behaviors in Multi-Agent Systems based on Multimodal Large Language Models](https://mlsanity.com/paper/2510.18515/) (2025)
- [CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation](https://mlsanity.com/paper/2510.19670/) (2025)
- [Human Machine Social Hybrid Intelligence:A Collaborative Decision Making Framework for Large Model Agent Groups and Human Experts](https://mlsanity.com/paper/2510.24030/) (2025)
- [ZK-SenseLM: Verifiable Large-Model Wireless Sensing with Selective Abstention and Zero-Knowledge Attestation](https://mlsanity.com/paper/2510.25677/) (2025)
- [Parameter-Efficient and Personalized Federated Training of Generative Models at the Edge](https://mlsanity.com/paper/2511.11585/) (2025)
- [Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning](https://mlsanity.com/paper/2606.22878/) (2026)

## Abstract (arXiv)

Federated fine-tuning (FedFT) provides an effective paradigm for fine-tuning large language models (LLMs) in privacy-sensitive scenarios. However, practical deployment remains challenging due to the limited resources on end devices. Existing methods typically utilize parameter-efficient fine-tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA), to substantially reduce communication overhead. Nevertheless, significant memory usage for activation storage and computational demands from full backpropagation remain major barriers to efficient deployment on resource-constrained end devices. Moreover, substantial resource heterogeneity across devices results in severe synchronization bottlenecks, diminishing the overall fine-tuning efficiency. To address these issues, we propose FedQuad, a novel LoRA-based FedFT framework that adaptively adjusts the LoRA depth (the number of consecutive tunable LoRA layers from the output) according to device computational capabilities, while employing activation quantization to reduce memory overhead, thereby enabling efficient deployment on resource-constrained devices. Specifically, FedQuad first identifies the feasible and efficient combinations of LoRA depth and the number of activation quantization layers based on device-specific resource constraints. Subsequently, FedQuad employs a greedy strategy to select the optimal configurations for each device, effectively accommodating system heterogeneity. Extensive experiments demonstrate that FedQuad achieves a 1.4-5.3x convergence acceleration compared to state-of-the-art baselines when reaching target accuracy, highlighting its efficiency and deployability in resource-constrained and heterogeneous end-device environments.