# Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

arXiv: https://arxiv.org/abs/2507.19839 · HTML: https://mlsanity.com/paper/2507.19839/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2025.

## Built on by (most cited first, 6 of 6)

- [Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting](https://mlsanity.com/paper/2508.04227/) (2025)
- [Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models](https://mlsanity.com/paper/2603.22094/) (2026)
- [Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models](https://mlsanity.com/paper/2512.13072/) (2025)
- [iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2605.19301/) (2026)
- [Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2609.13742/) (2026)
- [Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning](https://mlsanity.com/paper/2609.36759/) (2026)

## Abstract (arXiv)

The modality gap is intrinsic to multimodal alignment, where embeddings from different modalities remain partially separated in a shared representation space. In continual learning, CLIP-based vision-language models (VLMs) not only suffer catastrophic forgetting but also experience modality gap drift that disrupts the shared embedding geometry. Such drift degrades cross-modal alignment and generalization. We propose FPCD, an efficient framework combining Fast Null-space Projection (FNP) and Contrastive Distillation (CD). FNP accumulates compact Gram matrices of previous tasks' activations and projects gradients onto layer-adaptive approximate null spaces to reduce interference with learned representations. By replacing singular value decomposition with symmetric eigendecomposition, FNP accelerates projection-basis construction by a factor of 5-9. CD uses a small reference set to align the bidirectional image-to-text and text-to-image similarity distributions of the fine-tuned model with CLIP, thereby preserving pretrained cross-modal relations. On the 11-task MTIL benchmark, FPCD achieves 87.6% Last, 76.4% Average, and 65.2% Transfer. It also maintains more stable modality gaps and stronger zero-shot retrieval performance on COCO Captions and Flickr30k than competing methods. These results highlight the importance of controlling modality gap drift during continual adaptation of VLMs.