# What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignment

arXiv: https://arxiv.org/abs/2609.32485 · HTML: https://mlsanity.com/paper/2609.32485/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 19 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations](https://mlsanity.com/paper/2409.05976/) (2024)
- [Asymmetry in Low-Rank Adapters of Foundation Models](https://mlsanity.com/paper/2402.16842/) (2024)
- [Selective Aggregation for Low-Rank Adaptation in Federated Learning](https://mlsanity.com/paper/2410.01463/) (2024)
- [Dual-Personalizing Adapter for Federated Foundation Models](https://mlsanity.com/paper/2403.19211/) (2024)
- [CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2406.05223/) (2024)
- [LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement](https://mlsanity.com/paper/2411.14961/) (2024)
- [Closed-form merging of parameter-efficient modules for Federated Continual Learning](https://mlsanity.com/paper/2410.17961/) (2024)
- [Parameter Efficient Fine-tuning via Explained Variance Adaptation](https://mlsanity.com/paper/2410.07170/) (2024)
- [FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA](https://mlsanity.com/paper/2602.23638/) (2026)
- [Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs](https://mlsanity.com/paper/2509.25414/) (2025)

## Abstract (arXiv)

Federated low-rank adaptation (LoRA) requires identifying an update structure that is shared across heterogeneous clients. Prior work reports strong similarity among trained LoRA projection matrices across clients; however, such agreement may be largely induced by common initialization and collapses toward random overlap under independent initialization. More crucially, relying solely on parameter similarity inherently ignores the influence of local input regime. To uncover a more robust shared structure, we introduce an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs. Empirically, while parameter similarity vanishes, the leading right singular directions of this action matrix remain strongly aligned across clients. This shared geometry preserves task-conditioned differences and naturally varies across network depths. Motivated by these findings, we propose Federated Subspace-Guided Action-Informed Learning (FedSAIL). Instead of averaging weights, FedSAIL estimates a shared action subspace to regularize local training while preserving client-specific coefficients. Across several benchmarks, our approach consistently improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.