mlsanityData to 7 Oct 2026

AdaMerging: Adaptive Model Merging for Multi-Task Learning

arXiv 2310.02575 ↗First posted 4 Oct 2023cs.LG cs.CVcited by 248

Not on the map. Why?

Abstract (arXiv)

Multi-task learning (MTL) aims to empower a model to tackle multiple tasks simultaneously. A recent development known as task arithmetic has revealed that several models, each fine-tuned for distinct tasks, can be directly merged into a single model to execute MTL without necessitating a retraining process using the initial training data. Nevertheless, this direct addition of models often leads to a significant deterioration in the overall performance of the merged model. This decline occurs due to potential conflicts and intricate correlations among the multiple tasks. Consequently, the challenge emerges of how to merge pre-trained models more effectively without using their original training data. This paper introduces an innovative technique called Adaptive Model Merging (AdaMerging). This approach aims to autonomously learn the coefficients for model merging, either in a task-wise or layer-wise manner, without relying on the original training data. Specifically, our AdaMerging method operates as an automatic, unsupervised task arithmetic scheme. It leverages entropy minimization on unlabeled test samples from the multi-task setup as a surrogate objective function to iteratively refine the merging coefficients of the multiple models. Our experimental findings across eight tasks demonstrate the efficacy of the AdaMerging scheme we put forth. Compared to the current state-of-the-art task arithmetic merging scheme, AdaMerging showcases a remarkable 11% improvement in performance. Notably, AdaMerging also exhibits superior generalization capabilities when applied to unseen downstream tasks. Furthermore, it displays a significantly enhanced robustness to data distribution shifts that may occur during the testing phase.

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). Builds on 49 of its 87 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

49 of 87 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  11. 2014How transferable are features in deep neural networks?cited by 1,216
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
37 more
  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  3. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  4. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  6. 2020Federated Learning with Matched Averagingcited by 564
  7. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  8. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  9. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  10. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  11. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  12. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  13. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  14. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  15. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  16. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  17. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  18. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  19. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  20. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  21. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  22. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  23. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  24. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  25. 2020Learning to Branch for Multi-Task Learningcited by 114
  26. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  27. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113
  28. 2016Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classificationcited by 93
  29. 2023Deep Model Fusion: A Surveycited by 89
  30. 2019Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAScited by 83
  31. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  32. 2022Uncertainty-guided Source-free Domain Adaptationcited by 46
  33. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  34. 2022MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Taskscited by 34
  35. 2022AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learningcited by 31
  36. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  37. 2020Versatile Black-Box Optimizationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 53; 2025: 111; 2026: 82.

248 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  3. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  4. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  5. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  6. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  7. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  8. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  9. 2024Parameter Competition Balancing for Model Mergingcited by 71
  10. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  11. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 59
  12. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
38 more
  1. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  2. 2024What Matters for Model Merging at Scale?cited by 52
  3. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  4. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 49
  5. 2023Learning from models beyond fine-tuningcited by 47
  6. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  7. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  8. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  9. 2024Knowledge Composition using Task Vectors with Learned Anisotropic Scalingcited by 37
  10. 2023Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusioncited by 36
  11. 2025Model Merging in Pre-training of Large Language Modelscited by 36
  12. 2024Training-Free Pretrained Model Mergingcited by 32
  13. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  14. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  15. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  16. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  17. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
  18. 2024Revisiting Weight Averaging for Model Mergingcited by 24
  19. 2024Closed-form merging of parameter-efficient modules for Federated Continual Learningcited by 22
  20. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  21. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  22. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 22
  23. 2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20
  24. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  25. 2025FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectorscited by 19
  26. 2025Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learningcited by 19
  27. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  28. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 17
  29. 2024PLeaS -- Merging Models with Permutations and Least Squarescited by 16
  30. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  31. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  32. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  33. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  34. 2024Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmeticcited by 14
  35. 2025Nature-Inspired Population-Based Evolution of Large Language Modelscited by 14
  36. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  37. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  38. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 13

50 most cited of 248 All 248 (JSON)