# Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails

arXiv: https://arxiv.org/abs/2607.02020 · HTML: https://mlsanity.com/paper/2607.02020/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 25 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 22 of 22)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2303.06628/) (2023)
- [CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks](https://mlsanity.com/paper/2206.09059/) (2022)
- [CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model](https://mlsanity.com/paper/2403.08350/) (2024)
- [Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models](https://mlsanity.com/paper/2402.12048/) (2024)
- [Continual Vision-Language Representation Learning with Off-Diagonal Information](https://mlsanity.com/paper/2305.07437/) (2023)
- [CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering](https://mlsanity.com/paper/2503.00413/) (2025)
- [SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2505.02486/) (2025)
- [HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model](https://mlsanity.com/paper/2503.12941/) (2025)
- [ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt](https://mlsanity.com/paper/2410.05849/) (2024)
- [CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation](https://mlsanity.com/paper/2308.07146/) (2023)
- [MLLM-CL: Continual Learning for Multimodal Large Language Models](https://mlsanity.com/paper/2506.05453/) (2025)
- [Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models](https://mlsanity.com/paper/2411.02564/) (2024)
- [Synthetic Data is an Elegant GIFT for Continual Vision-Language Models](https://mlsanity.com/paper/2503.04229/) (2025)
- [Enhancing Multimodal Continual Instruction Tuning with BranchLoRA](https://mlsanity.com/paper/2506.02041/) (2025)
- [MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark](https://mlsanity.com/paper/2508.07307/) (2025)
- [LADA: Scalable Label-Specific CLIP Adapter for Continual Learning](https://mlsanity.com/paper/2505.23271/) (2025)
- [Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering](https://mlsanity.com/paper/2502.04469/) (2025)

## Built on by (most cited first, 1 of 1)

- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)

## Abstract (arXiv)

Multimodal large language models must continually adapt to evolving tasks and domains, yet standard continual learning metrics mainly measure whether old answers remain correct, leaving the stability of multimodal grounding largely unexamined. We study this overlooked failure mode and ask whether a continually adapted MLLM can preserve not only what it answers, but also how it uses visual, textual, OCR, chart, and document evidence. We identify hidden evidence-use forgetting, where answer accuracy is retained while the model silently shifts toward different or less grounded evidence channels, and propose textscRCL, a replay-free reliance-constrained continual learning framework. textscRCL freezes the previous checkpoint as a behavioral reference, estimates teacher and student evidence-reliance profiles through counterfactual channel interventions, and jointly optimizes task learning, prediction preservation, and reliance preservation without adding inference-time cost. Across CoIN, COAST, MCITlib, and an evidence-sensitive multimodal stream, textscRCL consistently improves final performance and reduces forgetting over replay-free, PEFT, routing, and memory-assisted baselines, while substantially lowering modality reliance drift, dominant evidence flips, and hidden forgetting rates. These results suggest that robust continual multimodal learning requires preserving the evidence path behind correct answers, not merely the answers themselves.