mlsanityData to 14 Oct 2026

Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs

arXiv 2505.16703 ↗First posted 22 May 2025cs.CLcited by 7

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Although multimodal large language models (MLLMs) have achieved impressive performance, the multimodal instruction tuning stage often causes catastrophic forgetting of the base LLM's language ability, even in strong models like Llama3. To address this, we propose Locate-then-Merge, a training-free parameter fusion framework that first locates important parameters and then selectively merges them. We further introduce Neuron-Fusion, a neuron-level strategy that preserves the influence of neurons with large parameter shifts--neurons likely responsible for newly acquired visual capabilities--while attenuating the influence of neurons with smaller changes that likely encode general-purpose language skills. This design enables better retention of visual adaptation while mitigating language degradation. Experiments on 13 benchmarks across both language and visual tasks show that Neuron-Fusion consistently outperforms existing model merging methods. Further analysis reveals that our method effectively reduces context hallucination in generation.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 52 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

39 of 52 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
27 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  6. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  8. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  10. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  11. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  12. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  13. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  14. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  15. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  16. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 97
  17. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  18. 2023Identifying and Mitigating Vulnerabilities in LLM-Integrated Applicationscited by 54
  19. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  20. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  21. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  22. 2024Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysiscited by 31
  23. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  24. 2025Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Modelscited by 23
  25. 2024TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidationcited by 18
  26. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 15
  27. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2026NEAT: Neuron-Based Early Exit for Large Reasoning Modelscited by 3
  3. 2025LEAML: Label-Efficient Adaptation to Out-of-Distribution Visual Tasks for Multimodal Large Language Modelscited by 0
  4. 2026Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuningcited by 0
  5. 2026VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulationcited by 0
  6. 2026NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMscited by 0
  7. 2026Decoupling Internal Representational Changes and Causal Importance in Fine-Tuned Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MAny: Merge Anything for Multimodal Continual Instruction Tuning
  2. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
  3. 2026SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuning
  4. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  5. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
  6. 2026AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
  7. 2024SwitchCIT: Switching for Continual Instruction Tuning
  8. 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning