When Do Task Vectors Interfere? Mapping the Validity Boundaries of Weight-Space Composition
Not on the map. Why?
Abstract (arXiv)
Task arithmetic composes skills by adding weight displacements, and merged models are then judged on benchmark suites. We measure when that composition is functionally additive, and find that the answer depends as much on how the model is prompted as on which tasks are merged. Across two-dimensional composition surfaces -- five model settings from 0.5B to 8B, two families, LoRA and full fine-tuning -- pairwise non-additivity is real, seed-stable, and transfers in coarse order to unseen task pairs: all eight preregistered sign predictions held. But it is input-conditioned everywhere we measured: the same merged model that shows a six-point interaction contrast on code prompts shows none on math prompts, and wrapping the identical code prompts in the instruction template the adapters were trained on collapses the contrast twenty-fold, from +6.9 to +0.3 points -- while re-serializing them in an untrained chat template leaves it intact (+12.5), falsifying our own preregistered prediction. Execution benchmarks (pass@1) inherit the training-format wrapper's blindness. Weight-space composition therefore supports coarse, input- and format-conditioned functional statements -- not a universal merging-performance predictor, and not one that training-format evaluations can see.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 10 Aug 2026.
Builds on
31 of 42 references.
References · 31
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
- 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
- 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
- 2021Merging Models with Fisher-Weighted Averagingcited by 613
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
19 more
- 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
- 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
- 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
- 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
- 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
- 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
- 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
- 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
- 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
- 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
- 2024Model merging with SVD to tie the Knotscited by 118
- 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
- 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
- 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
- 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
- 2023Merging by Matching Models in Task Parameter Subspacescited by 36
- 2024Merge to Learn: Efficiently Adding Skills to Language Models with Model Mergingcited by 7
- 2026Demystifying Mergeability: Interpretable Properties to Predict Model Merging Successcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.