mlsanityData to 7 Oct 2026

Superpose Task-specific Features for Model Merging

arXiv 2502.10698 ↗First posted 15 Feb 2025cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Model merging enables powerful capabilities in neural networks without requiring additional training. In this paper, we introduce a novel perspective on model merging by leveraging the fundamental mechanisms of neural network representation. Our approach is motivated by the linear representation hypothesis, which states that neural networks encode information through linear combinations of feature vectors. We propose a method that superposes task-specific features from individual models into a merged model. Our approach specifically targets linear transformation matrices, which are crucial for feature activation and extraction in deep networks. By formulating the merging process as a linear system, we can preserve task-specific features from individual models and create merged models that effectively maintain multi-task capabilities compared to existing methods. Extensive experiments across diverse benchmarks and models demonstrate that our method outperforms existing techniques. Code is available at https://github.com/LARS-research/STF.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 35 of its 62 references that are in the map. First posted on arXiv 15 Feb 2025.

Builds on

35 of 62 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
23 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  3. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  4. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  5. 2015Generating Sentences from a Continuous Spacecited by 607
  6. 2017Word Translation Without Parallel Datacited by 480
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  8. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  10. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  11. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  12. 2023In-Context Learning Creates Task Vectorscited by 302
  13. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  14. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  15. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  16. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  17. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  18. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  19. 2024Model merging with SVD to tie the Knotscited by 118
  20. 2022Fusing finetuned models for better pretrainingcited by 110
  21. 2019QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questionscited by 89
  22. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  23. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  2. 2026Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shiftscited by 0
  3. 2026Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approachcited by 0
  4. 2026Hierarchical Wasserstein Merging for Multi-Domain Multi-Task Learning: From Specialists to a Generalistcited by 0
  5. 2026A Survey on the Linear Representation Hypothesiscited by 0