mlsanityData to 7 Oct 2026

Parameter-Efficient Interventions for Enhanced Model Merging

arXiv 2412.17023 ↗First posted 22 Dec 2024cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Model merging combines knowledge from task-specific models into a unified multi-task model to avoid joint training on all task data. However, current methods face challenges due to representation bias, which can interfere with tasks performance. As a remedy, we propose IntervMerge, a novel approach to multi-task model merging that effectively mitigates representation bias across the model using taskspecific interventions. To further enhance its efficiency, we introduce mini-interventions, which modify only part of the representation, thereby reducing the additional parameters without compromising performance. Experimental results demonstrate that IntervMerge consistently outperforms the state-of-the-art approaches using fewer parameters.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

36 of 48 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
24 more
  1. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  2. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  3. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  4. 2020What is being transferred in transfer learning?cited by 365
  5. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  6. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  7. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  8. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  9. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  10. 2022Patching open-vocabulary models by interpolating weightscited by 187
  11. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  12. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  13. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  14. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  15. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113
  16. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  17. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  18. 2023Deep Model Fusion: A Surveycited by 89
  19. 2021Learning Neural Network Subspacescited by 82
  20. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  21. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  22. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  23. 2024Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Erascited by 26
  24. 2022Improving Generalization of Pre-trained Language Models via Stochastic Weight Averagingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  2. 2026FeatCal: Feature Calibration for Post-Merging Modelscited by 5
  3. 2026Representation Finetuning for Continual Learningcited by 0