mlsanityData to 14 Oct 2026

DPFormer: Dynamic Prompt Transformer for Continual Learning

arXiv 2506.07414 ↗First posted 9 Jun 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In continual learning, solving the catastrophic forgetting problem may make the models fall into the stability-plasticity dilemma. Moreover, inter-task confusion will also occur due to the lack of knowledge exchanges between different tasks. In order to solve the aforementioned problems, we propose a novel dynamic prompt transformer (DPFormer) with prompt schemes. The prompt schemes help the DPFormer memorize learned knowledge of previous classes and tasks, and keep on learning new knowledge from new classes and tasks under a single network structure with a nearly fixed number of model parameters. Moreover, they also provide discrepant information to represent different tasks to solve the inter-task confusion problem. Based on prompt schemes, a unified classification module with the binary cross entropy loss, the knowledge distillation loss and the auxiliary loss is proposed to train the whole model in an end-to-end trainable manner. Compared with state-of-the-art methods, our method achieves the best performance in the CIFAR-100, ImageNet100 and ImageNet1K datasets under different class-incremental settings in continual learning. The source code will be available at our GitHub after acceptance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

29 of 46 references.

References · 29

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2016Learning without Forgettingcited by 2,496
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  10. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  11. 2018Experience Replay for Continual Learningcited by 1,095
  12. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
17 more
  1. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  2. 2019Large Scale Incremental Learningcited by 732
  3. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  4. 2018End-to-End Incremental Learningcited by 597
  5. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  6. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  7. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  8. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  9. 2022Neighborhood Attention Transformercited by 239
  10. 2019Class-incremental Learning via Deep Model Consolidationcited by 173
  11. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  12. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
  13. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  14. 2022Dilated Neighborhood Attention Transformercited by 64
  15. 2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
  16. 2020Continual Learning with Extended Kronecker-factored Approximate Curvaturecited by 39
  17. 2021Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractorscited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation
  2. 2024Continual Distillation Learning for Rehearsal-Free Class-Incremental Learning via Decoupled Prompting
  3. 2024Remembering Transformer for Continual Learning
  4. 2024Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks
  5. 2024Mixture of Experts Meets Prompt-Based Continual Learning
  6. 2026GAP-Prompt: Gated Adaptive Prompting for Efficient Continual Learning
  7. 2024REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning
  8. 2025No Forgetting Learning: Buffer-free Continual Learning Classification