mlsanityData to 14 Oct 2026

CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

arXiv 2211.13218 ↗First posted 23 Nov 2022cs.CV cs.AI cs.LGcited by 450

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Computer vision models suffer from a phenomenon known as catastrophic forgetting when learning novel concepts from continuously shifting training data. Typical solutions for this continual learning problem require extensive rehearsal of previously seen data, which increases memory costs and may violate data privacy. Recently, the emergence of large-scale pre-trained vision transformer models has enabled prompting approaches as an alternative to data-rehearsal. These approaches rely on a key-query mechanism to generate prompts and have been found to be highly resistant to catastrophic forgetting in the well-established rehearsal-free continual learning setting. However, the key mechanism of these methods is not trained end-to-end with the task sequence. Our experiments show that this leads to a reduction in their plasticity, hence sacrificing new task accuracy, and inability to benefit from expanded parameter capacity. We instead propose to learn a set of prompt components which are assembled with input-conditioned weights to produce input-conditioned prompts, resulting in a novel attention-based end-to-end key-query scheme. Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 4.5% in average final accuracy. We also outperform the state of art by as much as 4.4% accuracy on a continual learning benchmark which contains both class-incremental and domain-incremental task shifts, corresponding to many practical settings. Our code is available at https://github.com/GT-RIPL/CODA-Prompt

Key facts. Cited by 450 papers in the map (data through 14 Oct 2026). Builds on 62 of its 78 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

62 of 78 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  9. 2016Learning without Forgettingcited by 2,496
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
38 more
  1. 2016Progressive Neural Networkscited by 1,361
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  3. 2017Continual Learning with Deep Generative Replaycited by 1,158
  4. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  5. 2018Experience Replay for Continual Learningcited by 1,095
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  7. 2021Learning to Prompt for Continual Learningcited by 944
  8. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  9. 2019Large Scale Incremental Learningcited by 732
  10. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  11. 2019Three scenarios for continual learningcited by 614
  12. 2018End-to-End Incremental Learningcited by 597
  13. 2019Gradient based sample selection for online continual learningcited by 571
  14. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  15. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  16. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
  17. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  18. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 308
  19. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  20. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  21. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 289
  22. 2019Continual learning with hypernetworkscited by 273
  23. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  24. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  25. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  26. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  27. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 182
  28. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  29. 2020Adversarial Continual Learningcited by 142
  30. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 134
  31. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 131
  32. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  33. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  34. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
  35. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111
  36. 2018Continuous Learning in Single-Incremental-Task Scenarioscited by 109
  37. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 104
  38. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 102

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 42; 2024: 140; 2025: 149; 2026: 119.

450 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2023Class-Incremental Learning: A Surveycited by 209
  3. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  4. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 184
  5. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 164
  6. 2024Continual Learning with Pre-Trained Models: A Surveycited by 154
  7. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  8. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 125
  9. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  10. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 117
  11. 2023PromptIR: Prompting for All-in-One Blind Image Restorationcited by 114
  12. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
38 more
  1. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  2. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  3. 2023PILOT: A Pre-Trained Model-Based Continual Learning Toolboxcited by 64
  4. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  5. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 63
  6. 2024Consistent Prompting for Rehearsal-Free Continual Learningcited by 54
  7. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  8. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47
  9. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  10. 2024SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Modelscited by 38
  11. 2024Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learnerscited by 36
  12. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Modelscited by 34
  13. 2024Resurrecting Old Classes with New Data for Exemplar-Free Continual Learningcited by 33
  14. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  15. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfercited by 31
  16. 2024Visual Prompt Tuning in Null Space for Continual Learningcited by 31
  17. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 30
  18. 2023Continual Learning with Low Rank Adaptationcited by 29
  19. 2024Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learningcited by 29
  20. 2023Learning to Modulate pre-trained Models in RLcited by 28
  21. 2025Language Guided Concept Bottleneck Models for Interpretable Continual Learningcited by 28
  22. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 28
  23. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 27
  24. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  25. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  26. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  27. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  28. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  29. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 23
  30. 2024Generative Multi-modal Models are Good Class-Incremental Learnerscited by 22
  31. 2024Adaptive Memory Replay for Continual Learningcited by 22
  32. 2024Closed-form merging of parameter-efficient modules for Federated Continual Learningcited by 22
  33. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  34. 2023Long-Tailed Continual Learning For Visual Food Recognitioncited by 20
  35. 2024OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learningcited by 20
  36. 2024Personalized Federated Continual Learning via Multi-granularity Promptcited by 20
  37. 2024HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuningcited by 20
  38. 2023Multimodal Parameter-Efficient Few-Shot Class Incremental Learningcited by 19

50 most cited of 450 All 450 (JSON)