mlsanityData to 14 Oct 2026

CrossMean: Model Merging by Pairing Merged and Expert Activations

arXiv 2610.14762 ↗First posted 10 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Model merging combines fine-tuned experts into one network without retraining, and closed-form methods compute each merged layer from statistics of its input activations. However, these statistics are measured on a single expert, while in the merged model, each layer receives activations from the merged layers that precede it. Recent work re-measures these statistics in the merged model but still forms them from a single network. We formulate closed-form methods as a linear system and show that they use the same matrix on both sides. We then introduce CrossMean, which replaces this matrix on one side with a bilinear statistic that pairs merged and expert activations. From the same statistics, we also derive a label-free criterion that indicates the task sets in which accounting for the merged activations improves the merge more. Empirically, task sets with a lower criterion value show a larger improvement. On the eight-task benchmark of each vision backbone that we tested and on BERT encoders, CrossMean improves on merging with statistics measured on the experts, and with all linear layers merged, it also improves on re-measuring these statistics in the merged model. These results suggest that merging benefits from pairing merged and expert activations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 60 of its 80 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

60 of 80 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
38 more
  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  3. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  4. 2021Robust fine-tuning of zero-shot modelscited by 797
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  8. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  9. 2018Deep Learning for Classical Japanese Literaturecited by 422
  10. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  11. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  12. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  13. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  14. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  15. 2019Model Fusion via Optimal Transportcited by 219
  16. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  17. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 188
  18. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  19. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  20. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  21. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  22. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  23. 2019Transferability and Hardness of Supervised Classification Taskscited by 127
  24. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  25. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  26. 2024Knowledge Fusion of Large Language Modelscited by 115
  27. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
  28. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  29. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  30. 2024Model Stock: All we need is just a few fine-tuned modelscited by 92
  31. 2022An Information-Theoretic Approach to Transferability in Task Transfer Learningcited by 81
  32. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  33. 2024Parameter Competition Balancing for Model Mergingcited by 72
  34. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 62
  35. 2024What Matters for Model Merging at Scale?cited by 55
  36. 2021Frustratingly Easy Transferability Estimationcited by 45
  37. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
  38. 2023Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusioncited by 37

50 most cited of 60 All 60 (JSON)

Built on by

Not cited yet.