mlsanityData to 14 Oct 2026

Curvature-Guided Mixing for MLLM Adaptation

arXiv 2606.24963 ↗First posted 23 Jun 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form "soft mixing" ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGMdagger, a robust "hard mixing" variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGMdagger consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

38 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model6 Mar
  2. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning17 Nov
  3. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoning26 Oct
  4. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training10 Oct
  5. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models19 Feb
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 31

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
19 more
  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  2. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  3. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  4. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  6. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  7. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  8. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  9. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 105
  10. 2022CLiMB: A Continual Learning Benchmark for Vision-and-Language Taskscited by 68
  11. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 63
  12. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 61
  13. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 30
  14. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  15. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  16. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  17. 2025PLAN: Proactive Low-Rank Allocation for Continual Learningcited by 7
  18. 2024Using Left and Right Brains Together: Towards Vision and Language Planningcited by 6
  19. 2025Hiding Images in Diffusion Models by Editing Learned Score Functionscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search
  2. 2025CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
  3. 2024CoMMIT: Coordinated Multimodal Instruction Tuning
  4. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  5. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging
  6. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
  7. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization
  8. 2026MoR-MLLM: Mixture of Recursions for Efficient Multimodal Large Language Models