mlsanityData to 7 Oct 2026

HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model

arXiv 2503.12941 ↗First posted 17 Mar 2025cs.CL cs.LGcited by 29

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Instruction tuning is widely used to improve a pre-trained Multimodal Large Language Model (MLLM) by training it on curated task-specific datasets, enabling better comprehension of human instructions. However, it is infeasible to collect all possible instruction datasets simultaneously in real-world scenarios. Thus, enabling MLLM with continual instruction tuning is essential for maintaining their adaptability. However, existing methods often trade off memory efficiency for performance gains, significantly compromising overall efficiency. In this paper, we propose a task-specific expansion and task-general fusion framework based on the variations in Centered Kernel Alignment (CKA) similarity across different model layers when trained on diverse datasets. Furthermore, we analyze the information leakage present in the existing benchmark and propose a new and more challenging benchmark to rationally evaluate the performance of different methods. Comprehensive experiments showcase a significant performance improvement of our method compared to existing state-of-the-art methods. Code and dataset are released at https://github.com/Ghy0501/HiDe-LLaVA.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 54 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

37 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Law of Vision Representation in MLLMs29 Aug
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2016Learning without Forgettingcited by 2,465
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
20 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2021Learning to Prompt for Continual Learningcited by 938
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  11. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  12. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  13. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  14. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  15. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
  16. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  17. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  18. 2023Learning from models beyond fine-tuningcited by 47
  19. 2023Learnability and Algorithm for Continual Learningcited by 28
  20. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 22.

29 papers.

Sort:

By area:

  1. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
  2. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  3. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  4. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 10
  5. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradientscited by 7
  6. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidancecited by 6
  7. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  8. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 4
  9. 2026Elastic Weight Consolidation Done Right for Continual Learningcited by 3
  10. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  11. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
  12. 2026Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuningcited by 2
17 more
  1. 2026UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Modelscited by 2
  2. 2025Towards Efficient and General-Purpose Few-Shot Misclassification Detection for Vision-Language Modelscited by 1
  3. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Thinkcited by 1
  4. 2026CL-VISTA: Benchmarking Continual Learning in Video Large Language Modelscited by 1
  5. 2026Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMscited by 1
  6. 2026Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Failscited by 1
  7. 2026CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detectioncited by 1
  8. 2026Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Modelscited by 0
  9. 2026ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editingcited by 0
  10. 2026Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Eracited by 0
  11. 2026KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptationcited by 0
  12. 2026InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memoriescited by 0
  13. 2026SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuningcited by 0
  14. 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuningcited by 0
  15. 2026UniAR: A Unified Framework for Autism Recognition Enhanced by Multi-View Prompt Learningcited by 0
  16. 2026ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuningcited by 0
  17. 2026Two Routes to the Middle: Placement Search and Brain Readouts Converge on Where Continual Learners Should Specializecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CoMMIT: Coordinated Multimodal Instruction Tuning
  2. 2024Towards Robust Instruction Tuning on Multimodal Large Language Models
  3. 2026LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning
  4. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
  5. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
  6. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
  7. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
  8. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models