mlsanityData to 14 Oct 2026

Model Merging with Functional Dual Anchors

arXiv 2510.21223 ↗First posted 24 Oct 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Model merging is an efficient post-training strategy for integrating knowledge from multiple finetuned checkpoints of a shared foundation model. Existing methods operate in the parameter space, combining task vectors to mitigate conflicts, but remain constrained by parameter inconsistencies. We propose Functional Dual Anchors (FDAs), a framework that instead models the input-representation space. FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pretrained model. This perspective bridges joint multi-task training and post-hoc merging, offering both robustness and flexibility. We further introduce a principled initialization scheme and show that FDAs are complementary to parameter-space model merging. Comprehensive experiments demonstrate the effectiveness of FDAs in model merging.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

52 of 67 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
38 more
  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  4. 2017Continual Learning with Deep Generative Replaycited by 1,158
  5. 2022Editing Models with Task Arithmeticcited by 1,120
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  11. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  12. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  13. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  14. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  15. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  16. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  17. 2020Dataset Condensation with Gradient Matchingcited by 516
  18. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  19. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  20. 2021Dataset Condensation with Distribution Matchingcited by 365
  21. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  22. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  23. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  24. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  25. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  26. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  27. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  28. 2024Parameter Competition Balancing for Model Mergingcited by 72
  29. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 62
  30. 2017Deep Hyperspherical Learningcited by 55
  31. 2018Decoupled Networkscited by 45
  32. 2017Iterative Machine Teachingcited by 41
  33. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  34. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficientlycited by 38
  35. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  36. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 13
  37. 2022Continual Learning by Modeling Intra-Class Variationcited by 12
  38. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  2. 2026Orthogonal Model Mergingcited by 4
  3. 2026PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspectivecited by 3
  4. 2026Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Mergingcited by 0