# Many-Task Federated Fine-Tuning via Unified Task Vectors

arXiv: https://arxiv.org/abs/2502.06376 · HTML: https://mlsanity.com/paper/2502.06376/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning](https://mlsanity.com/paper/2309.00363/) (2023)
- [Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models](https://mlsanity.com/paper/2401.06432/) (2024)
- [SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2308.06522/) (2023)
- [PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks](https://mlsanity.com/paper/2203.05126/) (2022)
- [Wasserstein Task Embedding for Measuring Task Similarities](https://mlsanity.com/paper/2208.11726/) (2022)
- [Gradient Masked Averaging for Federated Learning](https://mlsanity.com/paper/2201.11986/) (2022)
- [FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning](https://mlsanity.com/paper/2404.15182/) (2024)
- [MAS: Towards Resource-Efficient Federated Multiple-Task Learning](https://mlsanity.com/paper/2307.11285/) (2023)
- [FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering](https://mlsanity.com/paper/2404.15384/) (2024)
- [Federated Fine-Tuning of Foundation Models via Probabilistic Masking](https://mlsanity.com/paper/2311.17299/) (2023)

## Built on by (most cited first, 3 of 3)

- [Model Merging via Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2512.21288/) (2025)
- [EFU: Enforcing Federated Unlearning via Functional Encryption](https://mlsanity.com/paper/2508.07873/) (2025)
- [Collaborative Management for Chronic Diseases and Depression: A Double Heterogeneity-based Multi-Task Learning Method](https://mlsanity.com/paper/2511.16398/) (2025)

## Abstract (arXiv)

Federated Learning (FL) traditionally assumes homogeneous client tasks; however, in real-world scenarios, clients often specialize in diverse tasks, introducing task heterogeneity. To address this challenge, Many-Task FL (MaT-FL) has emerged, enabling clients to collaborate effectively despite task diversity. Existing MaT-FL approaches rely on client grouping or personalized layers, requiring the server to manage individual models and failing to account for clients handling multiple tasks. We propose MaTU, a MaT-FL approach that enables joint learning of task vectors across clients, eliminating the need for clustering or client-specific weight storage at the server. Our method introduces a novel aggregation mechanism that determines task similarity based on the direction of clients task vectors and constructs a unified task vector encapsulating all tasks. To address task-specific requirements, we augment the unified task vector with lightweight modulators that facilitate knowledge transfer among related tasks while disentangling dissimilar ones. Evaluated across 30 datasets, MaTU achieves superior performance over state-of-the-art MaT-FL approaches, with results comparable to per-task fine-tuning, while delivering significant communication savings.