# Consistent Prompting for Rehearsal-Free Continual Learning

arXiv: https://arxiv.org/abs/2403.08568 · HTML: https://mlsanity.com/paper/2403.08568/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 52 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 30 of 42)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Progressive Prompts: Continual Learning for Language Models](https://mlsanity.com/paper/2301.12314/) (2023)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality](https://mlsanity.com/paper/2310.07234/) (2023)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks](https://mlsanity.com/paper/2011.01844/) (2020)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)

## Built on by (most cited first, 30 of 54)

- [Visual Prompt Tuning in Null Space for Continual Learning](https://mlsanity.com/paper/2406.05658/) (2024)
- [Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2407.10281/) (2024)
- [Dynamic Integration of Task-Specific Adapters for Class Incremental Learning](https://mlsanity.com/paper/2409.14983/) (2024)
- [Componential Prompt-Knowledge Alignment for Domain Incremental Learning](https://mlsanity.com/paper/2505.04575/) (2025)
- [RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning](https://mlsanity.com/paper/2507.22553/) (2025)
- [Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning](https://mlsanity.com/paper/2508.01579/) (2025)
- [LW2G: Learning Whether to Grow for Prompt-based Continual Learning](https://mlsanity.com/paper/2409.18860/) (2024)
- [CAPrompt: Cyclic Prompt Aggregation for Pre-Trained Model Based Class Incremental Learning](https://mlsanity.com/paper/2412.08929/) (2024)
- [Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning](https://mlsanity.com/paper/2603.00191/) (2026)
- [LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning](https://mlsanity.com/paper/2410.00645/) (2024)
- [Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning](https://mlsanity.com/paper/2502.07560/) (2025)
- [Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning](https://mlsanity.com/paper/2505.22389/) (2025)
- [EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2506.12351/) (2025)
- [CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learning](https://mlsanity.com/paper/2507.09471/) (2025)
- [Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learning](https://mlsanity.com/paper/2411.15469/) (2024)
- [Achieving More with Less: Additive Prompt Tuning for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2503.07979/) (2025)
- [KAC: Kolmogorov-Arnold Classifier for Continual Learning](https://mlsanity.com/paper/2503.21076/) (2025)
- [Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2407.17120/) (2024)
- [Leveraging Hierarchical Taxonomies in Prompt-based Continual Learning](https://mlsanity.com/paper/2410.04327/) (2024)
- [Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation](https://mlsanity.com/paper/2505.11998/) (2025)
- [Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2506.12409/) (2025)
- [Choice of PEFT Technique in Continual Learning: Prompt Tuning is Not All You Need](https://mlsanity.com/paper/2406.03216/) (2024)
- [Not Just Object, But State: Compositional Incremental Learning without Forgetting](https://mlsanity.com/paper/2411.01739/) (2024)
- [Continual Learning for Segment Anything Model Adaptation](https://mlsanity.com/paper/2412.06418/) (2024)
- [Adapter-Enhanced Semantic Prompting for Continual Learning](https://mlsanity.com/paper/2412.11074/) (2024)
- [DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognition](https://mlsanity.com/paper/2502.00618/) (2025)
- [Distribution-aware Forgetting Compensation for Exemplar-Free Lifelong Person Re-identification](https://mlsanity.com/paper/2504.15041/) (2025)
- [One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning](https://mlsanity.com/paper/2509.24483/) (2025)
- [InTAct: Interval-based Task Activation Consolidation for Continual Learning](https://mlsanity.com/paper/2511.17439/) (2025)
- [Deep Lidar-guided Image Deblurring](https://mlsanity.com/paper/2412.07262/) (2024)

## Abstract (arXiv)

Continual learning empowers models to adapt autonomously to the ever-changing environment or data streams without forgetting old knowledge. Prompt-based approaches are built on frozen pre-trained models to learn the task-specific prompts and classifiers efficiently. Existing prompt-based methods are inconsistent between training and testing, limiting their effectiveness. Two types of inconsistency are revealed. Test predictions are made from all classifiers while training only focuses on the current task classifier without holistic alignment, leading to Classifier inconsistency. Prompt inconsistency indicates that the prompt selected during testing may not correspond to the one associated with this task during training. In this paper, we propose a novel prompt-based method, Consistent Prompting (CPrompt), for more aligned training and testing. Specifically, all existing classifiers are exposed to prompt training, resulting in classifier consistency learning. In addition, prompt consistency learning is proposed to enhance prediction robustness and boost prompt selection accuracy. Our Consistent Prompting surpasses its prompt-based counterparts and achieves state-of-the-art performance on multiple continual learning benchmarks. Detailed analysis shows that improvements come from more consistent training and testing.