mlsanityData to 14 Oct 2026

Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing

arXiv 2606.16501 ↗First posted 15 Jun 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Model merging has become a practical post-training strategy for building a single multi-task large language model (LLM) by combining multiple task-specialized models. However, most existing approaches rely on post-hoc merging, in which task-specific models are merged only once after training. This one-shot aggregation often suffers from task interference, leading to information erasure across individual tasks. In this work, we show that replacing post-hoc merging with an iterative many-shot merging protocol is effective in improving multi-task performance. Building on this insight, we propose METIS, Mitigating Erasure from Task Interference for Stable many-shot merging. METIS is a loss-aware many-shot merging method that addresses information erasure in post-hoc merging through task-wise loss-gap weighting and consensus-based masking. Notably, METIS exhibits significant performance improvement on the worst-performing task, effectively mitigating information erasure. (Project page: https://imkyungjin.github.io/METIS/)

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

50 of 65 references.

References · 50

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2021Convex optimizationcited by 2,515
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2022Editing Models with Task Arithmeticcited by 1,120
38 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  3. 2021Robust fine-tuning of zero-shot modelscited by 797
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  5. 2018CoQA: A Conversational Question Answering Challengecited by 683
  6. 2017Federated Multi-Task Learningcited by 672
  7. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  8. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  10. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  11. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  12. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  13. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  14. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  15. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  16. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  17. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  18. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  19. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  20. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  21. 2022Fusing finetuned models for better pretrainingcited by 110
  22. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  23. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  24. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
  25. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  26. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  27. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 33
  28. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  29. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 23
  30. 2022AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimationcited by 21
  31. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  32. 2025Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integrationcited by 13
  33. 2025Subspace-Boosted Model Mergingcited by 9
  34. 2025Merge-of-Thought Distillationcited by 7
  35. 2025FedMerge: Federated Personalization via Model Mergingcited by 4
  36. 2025ASAP: Unsupervised Post-training with Label Distribution Shift Adaptive Learning Ratecited by 3
  37. 2026SToRM: Supervised Token Reduction for Multi-modal LLMs toward efficient end-to-end autonomous drivingcited by 3
  38. 2026Fed-ADE: Adaptive Learning Rate for Federated Post-adaptation under Distribution Shiftcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SuperAI: Merging Capabilities Across Heterogeneous Foundation Modelscited by 0