mlsanityData to 14 Oct 2026

Bi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approach

arXiv 2509.12697 ↗First posted 16 Sep 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated foundation models represent a new paradigm to jointly fine-tune pre-trained foundation models across clients. It is still a challenge to fine-tune foundation models for a small group of new users or specialized scenarios, which typically involve limited data compared to the large-scale data used in pre-training. In this context, the trade-off between personalization and federation becomes more sensitive. To tackle these, we proposed a bi-level personalization framework for federated fine-tuning on foundation models. Specifically, we conduct personalized fine-tuning on the client-level using its private data, and then conduct a personalized aggregation on the server-level using similar users measured by client-specific task vectors. Given the personalization information gained from client-level fine-tuning, the server-level personalized aggregation can gain group-wise personalization information while mitigating the disturbance of irrelevant or interest-conflict clients with non-IID data. The effectiveness of the proposed algorithm has been demonstrated by extensive experimental analysis in benchmark datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

23 of 31 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  8. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  9. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  10. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 283
  11. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  12. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
11 more
  1. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  2. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
  3. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  4. 2021Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learningcited by 53
  5. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 50
  6. 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 36
  7. 2023FedSecurity: Benchmarking Attacks and Defenses in Federated Learning and Federated LLMscited by 21
  8. 2025FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectorscited by 19
  9. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  10. 2024Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integrationcited by 5
  11. 2025Federated Adapter on Foundation Models: An Out-Of-Distribution Approachcited by 2

Built on by

Not cited yet.