mlsanityData to 7 Oct 2026

Task Arithmetic Through The Lens Of One-Shot Federated Learning

arXiv 2411.18607 ↗First posted 27 Nov 2024cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Task Arithmetic is a model merging technique that enables the combination of multiple models' capabilities into a single model through simple arithmetic in the weight space, without the need for additional fine-tuning or access to the original training data. However, the factors that determine the success of Task Arithmetic remain unclear. In this paper, we examine Task Arithmetic for multi-task learning by framing it as a one-shot Federated Learning problem. We demonstrate that Task Arithmetic is mathematically equivalent to the commonly used algorithm in Federated Learning, called Federated Averaging (FedAvg). By leveraging well-established theoretical results from FedAvg, we identify two key factors that impact the performance of Task Arithmetic: data heterogeneity and training heterogeneity. To mitigate these challenges, we adapt several algorithms from Federated Learning to improve the effectiveness of Task Arithmetic. Our experiments demonstrate that applying these algorithms can often significantly boost performance of the merged model compared to the original Task Arithmetic approach. This work bridges Task Arithmetic and Federated Learning, offering new theoretical perspectives on Task Arithmetic and improved practical methodologies for model merging.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 81 of its 91 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

81 of 91 references.

References · 81

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2019Advances and Open Problems in Federated Learningcited by 3,021
  11. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  12. 2013Describing Textures in the Wildcited by 2,110
38 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2022Editing Models with Task Arithmeticcited by 1,076
  10. 2020Adaptive Federated Optimizationcited by 1,036
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  13. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  14. 2021Robust fine-tuning of zero-shot modelscited by 785
  15. 2021Model-Contrastive Federated Learningcited by 762
  16. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  17. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  18. 2017Federated Multi-Task Learningcited by 669
  19. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  20. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  21. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  22. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  23. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  24. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  25. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  26. 2021Towards Personalized Federated Learningcited by 462
  27. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  28. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  29. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  30. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  31. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  32. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  33. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  34. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  35. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  36. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  37. 2024Mixture of LoRA Expertscited by 179
  38. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 2.

12 papers.

  1. 2025Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Mergingcited by 18
  2. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  3. 2025FedMerge: Federated Personalization via Model Mergingcited by 4
  4. 2025Towards One-shot Federated Learning: Advances, Challenges, and Future Directionscited by 2
  5. 2025Model Merging via Multi-Teacher Knowledge Distillationcited by 2
  6. 2025Statistical Deficiency for Task Inclusion Estimationcited by 1
  7. 2025FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCAcited by 1
  8. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Expertscited by 1
  9. 2025Federated Independent Component Analysis via Spectral Alignment and Robust Aggregationcited by 0
  10. 2025Bi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approachcited by 0
  11. 2026Ecologically-Constrained Task Arithmetic for Multi-Taxa Bioacoustic Classifiers Without Shared Datacited by 0
  12. 2026Spherical Interpolation for Backward-Compatible Multimodal Representationscited by 0