# HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learning

arXiv: https://arxiv.org/abs/2603.02785 · HTML: https://mlsanity.com/paper/2603.02785/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Builds on (most cited first, 30 of 36)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Federated Multi-Task Learning under a Mixture of Distributions](https://mlsanity.com/paper/2108.10252/) (2021)
- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [FedBABU: Towards Enhanced Representation for Federated Image Classification](https://mlsanity.com/paper/2106.06042/) (2021)
- [FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction](https://mlsanity.com/paper/2203.11751/) (2022)
- [FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations](https://mlsanity.com/paper/2409.05976/) (2024)
- [Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources](https://mlsanity.com/paper/2402.11505/) (2024)
- [When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions](https://mlsanity.com/paper/2306.15546/) (2023)
- [Selective Aggregation for Low-Rank Adaptation in Federated Learning](https://mlsanity.com/paper/2410.01463/) (2024)
- [FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning](https://mlsanity.com/paper/2308.12305/) (2023)
- [FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning](https://mlsanity.com/paper/2406.07925/) (2024)
- [Dual-Personalizing Adapter for Federated Foundation Models](https://mlsanity.com/paper/2403.19211/) (2024)
- [Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation](https://mlsanity.com/paper/2308.15367/) (2023)
- [FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure](https://mlsanity.com/paper/2010.06870/) (2020)
- [pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning](https://mlsanity.com/paper/2310.13283/) (2023)
- [Personalized Federated Learning under Mixture of Distributions](https://mlsanity.com/paper/2305.01068/) (2023)
- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [An Aggregation-Free Federated Learning for Tackling Data Heterogeneity](https://mlsanity.com/paper/2404.18962/) (2024)
- [Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning](https://mlsanity.com/paper/2211.07864/) (2022)
- [FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averaging](https://mlsanity.com/paper/2507.20016/) (2025)
- [FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learning](https://mlsanity.com/paper/2308.09160/) (2023)
- [Ten Challenging Problems in Federated Foundation Models](https://mlsanity.com/paper/2502.12176/) (2025)
- [Confidence-aware Personalized Federated Learning via Variational Expectation Maximization](https://mlsanity.com/paper/2305.12557/) (2023)

## Abstract (arXiv)

Vision Transformers (ViTs) have been widely adopted in vision tasks due to their strong transferability. In Federated Learning (FL), where full fine-tuning is communication heavy, Low-Rank Adaptation (LoRA) provides an efficient and communication-friendly way to adapt ViTs. However, existing LoRA-based federated tuning methods overlook latent client structures in real-world settings, limiting shared representation learning and hindering effective adaptation to unseen clients. To address this, we propose HiLoRA, a hierarchical LoRA framework that places adapters at three levels: root, cluster, and leaf, each designed to capture global, subgroup, and client-specific knowledge, respectively. Through cross-tier orthogonality and cascaded optimization, HiLoRA separates update subspaces and aligns each tier with its residual personalized objective. In particular, we develop a LoRA-Subspace Adaptive Clustering mechanism that infers latent client groups via subspace similarity analysis, thereby facilitating knowledge sharing across structurally aligned clients. Theoretically, we establish a tier-wise generalization analysis that supports HiLoRA's design. Experiments on ViT backbones with CIFAR-100 and DomainNet demonstrate consistent improvements in both personalization and generalization.