# DPFormer: Dynamic Prompt Transformer for Continual Learning

arXiv: https://arxiv.org/abs/2506.07414 · HTML: https://mlsanity.com/paper/2506.07414/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 29 of 29)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Neighborhood Attention Transformer](https://mlsanity.com/paper/2204.07143/) (2022)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting](https://mlsanity.com/paper/1802.02950/) (2018)
- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Dilated Neighborhood Attention Transformer](https://mlsanity.com/paper/2209.15001/) (2022)
- [Efficient Feature Transformations for Discriminative and Generative Continual Learning](https://mlsanity.com/paper/2103.13558/) (2021)
- [Continual Learning with Extended Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/2004.07507/) (2020)
- [Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors](https://mlsanity.com/paper/2104.13614/) (2021)

## Abstract (arXiv)

In continual learning, solving the catastrophic forgetting problem may make the models fall into the stability-plasticity dilemma. Moreover, inter-task confusion will also occur due to the lack of knowledge exchanges between different tasks. In order to solve the aforementioned problems, we propose a novel dynamic prompt transformer (DPFormer) with prompt schemes. The prompt schemes help the DPFormer memorize learned knowledge of previous classes and tasks, and keep on learning new knowledge from new classes and tasks under a single network structure with a nearly fixed number of model parameters. Moreover, they also provide discrepant information to represent different tasks to solve the inter-task confusion problem. Based on prompt schemes, a unified classification module with the binary cross entropy loss, the knowledge distillation loss and the auxiliary loss is proposed to train the whole model in an end-to-end trainable manner. Compared with state-of-the-art methods, our method achieves the best performance in the CIFAR-100, ImageNet100 and ImageNet1K datasets under different class-incremental settings in continual learning. The source code will be available at our GitHub after acceptance.