mlsanityData to 14 Oct 2026

PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs

arXiv 2601.07645 ↗First posted 12 Jan 2026cs.CLcited by 3

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining multimodal performance. To address this issue, we propose a training-free framework to mitigate this degradation. Through layer-wise vision token masking, we reveal a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and late-modal degradation. By analyzing the behavior of MLLMs at different stages, we propose a plateau-guided model merging method that selectively injects base language model parameters into MLLMs. Experimental results based on five MLLMs on nine benchmarks demonstrate the effectiveness of our method. Attention-based analysis further reveals that merging shifts attention from diffuse, scattered patterns to focused localization on task-relevant visual regions. Our repository is on https://github.com/wzj1718/PlaM.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 36 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

22 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Mitigating Modal Imbalance in Multimodal Reasoning2 Oct
  2. 2024Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning4 Dec

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  10. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  11. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  12. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
8 more
  1. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  2. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  3. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  4. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  5. 2025Vision Function Layer in Multimodal LLMscited by 17
  6. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  7. 2025A Survey of Generative Categories and Techniques in Multimodal Generative Modelscited by 4
  8. 2024Acquiring Linguistic Knowledge from Multimodal Inputcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  2. 2026NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMscited by 0
  3. 2026Orthogonal Yet Coupled: Decoupling Geometric Components for Model Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  2. 2025PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs
  3. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  4. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging
  5. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  6. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  7. 2025Visual Representation Alignment for Multimodal Large Language Models
  8. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings