# Visual Prompt Tuning in Null Space for Continual Learning

arXiv: https://arxiv.org/abs/2406.05658 · HTML: https://mlsanity.com/paper/2406.05658/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 9 Jun 2024.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [RanPAC: Random Projections and Pre-trained Models for Continual Learning](https://mlsanity.com/paper/2307.02251/) (2023)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [A Unified Continual Learning Framework with General Parameter-Efficient Tuning](https://mlsanity.com/paper/2303.10070/) (2023)
- [Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality](https://mlsanity.com/paper/2310.07234/) (2023)
- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [Continual Learning in Low-rank Orthogonal Subspaces](https://mlsanity.com/paper/2010.11635/) (2020)
- [Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning](https://mlsanity.com/paper/2110.04593/) (2021)
- [Consistent Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2403.08568/) (2024)
- [Introducing Language Guidance in Prompt-based Continual Learning](https://mlsanity.com/paper/2308.15827/) (2023)
- [AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learning](https://mlsanity.com/paper/2305.11488/) (2023)
- [Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference](https://mlsanity.com/paper/2211.15969/) (2022)
- [When Prompt-based Incremental Learning Does Not Meet Strong Pretraining](https://mlsanity.com/paper/2308.10445/) (2023)

## Built on by (most cited first, 30 of 31)

- [MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2412.09441/) (2024)
- [SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models](https://mlsanity.com/paper/2503.07392/) (2025)
- [Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning](https://mlsanity.com/paper/2410.00911/) (2024)
- [Parameter-Efficient Continual Fine-Tuning: A Survey](https://mlsanity.com/paper/2504.13822/) (2025)
- [SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting](https://mlsanity.com/paper/2505.22370/) (2025)
- [Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning](https://mlsanity.com/paper/2508.01579/) (2025)
- [Advancing Prompt-Based Methods for Replay-Independent General Continual Learning](https://mlsanity.com/paper/2503.00677/) (2025)
- [When Continue Learning Meets Multimodal Large Language Model: A Survey](https://mlsanity.com/paper/2503.01887/) (2025)
- [Continuous Subspace Optimization for Continual Learning](https://mlsanity.com/paper/2505.11816/) (2025)
- [Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning](https://mlsanity.com/paper/2603.00191/) (2026)
- [Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning](https://mlsanity.com/paper/2502.07560/) (2025)
- [Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning](https://mlsanity.com/paper/2505.22389/) (2025)
- [Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector](https://mlsanity.com/paper/2502.05540/) (2025)
- [EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2506.12351/) (2025)
- [CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learning](https://mlsanity.com/paper/2507.09471/) (2025)
- [Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection](https://mlsanity.com/paper/2602.07892/) (2026)
- [Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learning](https://mlsanity.com/paper/2411.15469/) (2024)
- [Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2407.17120/) (2024)
- [Replay-Free Continual Low-Rank Adaptation with Dynamic Memory](https://mlsanity.com/paper/2411.00623/) (2024)
- [Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts](https://mlsanity.com/paper/2602.03473/) (2026)
- [DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognition](https://mlsanity.com/paper/2502.00618/) (2025)
- [YOLO-IOD: Towards Real Time Incremental Object Detection](https://mlsanity.com/paper/2512.22973/) (2025)
- [Dance Across Shifts: Forward-Facilitation Continual Test-Time Adaptation through Dynamic Style Bridging](https://mlsanity.com/paper/2605.18608/) (2026)
- [Frugal Incremental Generative Modeling using Variational Autoencoders](https://mlsanity.com/paper/2505.22408/) (2025)
- [Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models](https://mlsanity.com/paper/2510.21175/) (2025)
- [Deep Generative Continual Learning using Functional LoRA: FunLoRA](https://mlsanity.com/paper/2510.02631/) (2025)
- [Scalable Analytic Classifiers with Associative Drift Compensation for Class-Incremental Learning of Vision Transformers](https://mlsanity.com/paper/2602.00144/) (2026)
- [Attention Retention for Continual Learning with Vision Transformers](https://mlsanity.com/paper/2602.05454/) (2026)
- [SR2-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning](https://mlsanity.com/paper/2605.07420/) (2026)
- [iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2605.19301/) (2026)

## Abstract (arXiv)

Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL), by selecting and updating relevant prompts in the vision-transformer models. On the contrary, this paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as to ensure no interference on tasks that have been learned to overcome catastrophic forgetting in CL. However, different from the orthogonal projection in the traditional CNN architecture, the prompt gradient orthogonal projection in the ViT architecture shows completely different and greater challenges, i.e., 1) the high-order and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. Theoretically, we have finally deduced two consistency conditions to achieve the prompt gradient orthogonal projection, which provide a theoretical guarantee of eliminating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient orthogonal projection. Extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods. Our code is available at https://github.com/zugexiaodui/VPTinNSforCL.