Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation
Not on the map. Why?
Abstract (arXiv)
Model merging enables training-free multi-task learning by aggregating knowledge from models fine-tuned from a shared checkpoint, but their task updates can interfere. Using proxy data, we find that task updates induce output shifts whose energy concentrates in a few dominant directions, while many low-energy directions can accumulate across tasks and impair performance. We call the span of the dominant directions the essential subspace. This observation motivates a data-aware framework to preserve essential task knowledge while reducing interference. Essential Subspace Decomposition (ESD) identifies each task's essential subspace from activation shifts measured on proxy data and represents its update compactly while preserving functional behavior. Essential Subspace Merging (ESM) weights each task-update direction by its corresponding output-shift eigenvalue, thereby prioritizing core task knowledge; it then orthogonalizes the weighted directions across tasks to reduce interference. ESM++ extends this merged model by dynamically integrating low-rank task-specific residual experts for adaptive knowledge aggregation. Experiments across diverse task sets and model scales show that ESM and ESM++ effectively aggregate task knowledge while mitigating inter-task interference. Code is available at https://github.com/kiddo127/ESM.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 62 of its 77 references that are in the map. First posted on arXiv 17 Jun 2026.
Builds on
62 of 77 references.
References · 62
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2016Pointer Sentinel Mixture Modelscited by 3,096
38 more
- 2013Describing Textures in the Wildcited by 2,125
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2022Editing Models with Task Arithmeticcited by 1,120
- 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
- 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
- 2021Merging Models with Fisher-Weighted Averagingcited by 621
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
- 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
- 2018Deep Learning for Classical Japanese Literaturecited by 422
- 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
- 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
- 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
- 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
- 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
- 2019Model Fusion via Optimal Transportcited by 219
- 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
- 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
- 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
- 2024Model merging with SVD to tie the Knotscited by 122
- 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
- 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
- 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
- 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 86
- 2024Parameter Competition Balancing for Model Mergingcited by 72
- 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 67
- 2022Re-basin via implicit Sinkhorn differentiationcited by 62
- 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 62
- 2022Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuningcited by 50
- 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
- 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
- 2024Knowledge Composition using Task Vectors with Learned Anisotropic Scalingcited by 37
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.