mlsanityData to 7 Oct 2026

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning

arXiv 2505.02486 ↗First posted 5 May 2025cs.LG cs.AIcited by 32

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Multimodal Continual Instruction Tuning (MCIT) aims to enable Multimodal Large Language Models (MLLMs) to incrementally learn new tasks without catastrophic forgetting. In this paper, we explore forgetting in this context, categorizing it into superficial forgetting and essential forgetting. Superficial forgetting refers to cases where the model's knowledge may not be genuinely lost, but its responses to previous tasks deviate from expected formats due to the influence of subsequent tasks' answer styles, making the results unusable. By contrast, essential forgetting refers to situations where the model provides correctly formatted but factually inaccurate answers, indicating a true loss of knowledge. Assessing essential forgetting necessitates addressing superficial forgetting first, as severe superficial forgetting can obscure the model's knowledge state. Hence, we first introduce the Answer Style Diversification (ASD) paradigm, which defines a standardized process for transforming data styles across different tasks, unifying their training sets into similarly diversified styles to prevent superficial forgetting caused by style shifts. Building on this, we propose RegLoRA to mitigate essential forgetting. RegLoRA stabilizes key parameters where prior knowledge is primarily stored by applying regularization, enabling the model to retain existing competencies. Experimental results demonstrate that our overall method, SEFE, achieves state-of-the-art performance.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 73 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

49 of 73 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models4 Nov
  2. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model13 Mar
  3. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer17 Jan
  4. 2023Continual Instruction Tuning for Large Multimodal Models27 Nov

Other references · 45

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
33 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  3. 2021Learning to Prompt for Continual Learningcited by 938
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  6. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  7. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  11. 2018nocaps: novel object captioning at scalecited by 444
  12. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  13. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  14. 2018Learning without Memorizingcited by 249
  15. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  16. 2019Episodic Memory in Lifelong Language Learningcited by 188
  17. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 181
  18. 2024Continual Learning for Large Language Models: A Surveycited by 144
  19. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  20. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  21. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  22. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  23. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  24. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  25. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  26. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  27. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  28. 2025Spurious Forgetting in Continual Learning of Language Modelscited by 47
  29. 2024SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Modelscited by 37
  30. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  31. 2021Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Maskingcited by 22
  32. 2024Towards Effective and Efficient Continual Pre-training of Large Language Modelscited by 13
  33. 2024Strike a Balance in Continual Panoptic Segmentationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 24.

32 papers.

Sort:

By area:

  1. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  2. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  3. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 10
  4. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  5. 2025OASIS: Online Sample Selection for Continual Visual Instruction Tuningcited by 8
  6. 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuningcited by 8
  7. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidancecited by 6
  8. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 4
  9. 2025Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMscited by 3
  10. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  11. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  12. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
20 more
  1. 2026Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuningcited by 2
  2. 2026RL Forgets! Towards Continual Policy Optimizationcited by 2
  3. 2026CL-VISTA: Benchmarking Continual Learning in Video Large Language Modelscited by 1
  4. 2026Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMscited by 1
  5. 2026Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Failscited by 1
  6. 2025Routing-Based Continual Learning for Multimodal Large Language Modelscited by 0
  7. 2026Continual-NExT: A Unified Comprehension And Generation Continual Learning Frameworkcited by 0
  8. 2026Data Organization Matters in Multimodal Instruction Tuning: A Controlled Study of Capability Trade-offscited by 0
  9. 2026Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysiscited by 0
  10. 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuningcited by 0
  11. 2026ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuningcited by 0
  12. 2026KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptationcited by 0
  13. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMscited by 0
  14. 2026InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memoriescited by 0
  15. 2026RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injectioncited by 0
  16. 2026DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuningcited by 0
  17. 2026A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Trainingcited by 0
  18. 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuningcited by 0
  19. 2026Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuningcited by 0
  20. 2026ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MAny: Merge Anything for Multimodal Continual Instruction Tuning
  2. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
  3. 2024SwitchCIT: Switching for Continual Instruction Tuning
  4. 2026AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
  5. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
  6. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
  7. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning
  8. 2026Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptation