Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptation

With the rapid emergence of foundation models and the increasing need for fine-tuning across distributed environments, Federated Low-Rank Adaptation (FedLoRA) has recently gained significant attention. Despite enormous potential, current FedLoRA methods face notable challenges due to inexact updates. Existing approaches have attempted to mitigate this issue, but they often introduce a local-global generalization gap and incur substantial communication overhead, limiting their scalability and effectiveness. To address these limitations, we propose Federated Low-Rank Aggregation with Nearly Accurate Estimation (FLoRA-NA). FLoRA-NA leverages the local LoRA matrices on the server to estimate the aggregated matrices hatA and hatB, which are then distributed to clients for local updates. This surrogated aggregated matrices minimizes the divergence between ideal nabla BarW = sumUu=1Bu Au and practical updates nabla hatW = hatBhatA without adding communication cost beyond vanilla FedLoRA. By doing so, FLoRA-NA achieves communication efficiency and bridges the gap between local personalization and global generalization, addressing a key limitation of prior personalized FedLoRA approaches. We conduct extensive evaluations across diverse tasks, including natural language understanding, mathematical reasoning, and code-solving ability using various foundation models. Experimental results consistently demonstrate that FLoRA-NA achieves state-of-the-art global performance while maintaining low communication overhead.