# Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning

arXiv: https://arxiv.org/abs/2410.00911 · HTML: https://mlsanity.com/paper/2410.00911/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 93 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 30 of 63)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)

## Built on by (most cited first, 16 of 16)

- [Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Experts](https://mlsanity.com/paper/2507.07100/) (2025)
- [External Knowledge Injection for CLIP-Based Class-Incremental Learning](https://mlsanity.com/paper/2503.08510/) (2025)
- [BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning](https://mlsanity.com/paper/2511.11421/) (2025)
- [Energy-Structured Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2605.27482/) (2026)
- [Sparse Orthogonal Parameters Tuning for Continual Learning](https://mlsanity.com/paper/2411.02813/) (2024)
- [Continual Knowledge Consolidation LORA for Domain Incremental Learning](https://mlsanity.com/paper/2510.16077/) (2025)
- [Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2605.09640/) (2026)
- [C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning](https://mlsanity.com/paper/2508.18860/) (2025)
- [SphereVideo: Prototype-anchored Hyperspherical Boundary for Continual AI-generated Video Detection](https://mlsanity.com/paper/2608.01334/) (2026)
- [ContiGuard: A Framework for Continual Toxicity Detection Against Evolving Evasive Perturbations](https://mlsanity.com/paper/2603.14843/) (2026)
- [Robust Embodied Perception in Dynamic Environments via Disentangled Weight Fusion](https://mlsanity.com/paper/2604.01669/) (2026)
- [Few-Shot Domain Incremental Learning via Continual Vision-Language Consolidation](https://mlsanity.com/paper/2606.30190/) (2026)
- [Task-Anchored Representation Shaping for Pre-Trained Model-Based Continual Learning](https://mlsanity.com/paper/2608.16345/) (2026)
- [Restoring Without Forgetting: Continual Learning Across Image Degradations](https://mlsanity.com/paper/2608.23799/) (2026)
- [Mitigating Performance Discrepancy in Cross-Domain 3D Class-Incremental Learning](https://mlsanity.com/paper/2609.04860/) (2026)
- [Online Versatile Incremental Learning: Towards Class and Domain-Agnostic Adaptation at Any Time](https://mlsanity.com/paper/2609.36442/) (2026)

## Abstract (arXiv)

Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. While recent advances in pre-trained models provide a solid foundation for DIL, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. Specifically, sequential model updates can overwrite both the representation and the classifier with knowledge from the latest domain. Thus, it is crucial to develop a representation and corresponding classifier that accommodate all seen domains throughout the learning process. To this end, we propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge at both the representation and classifier levels. By merging the backbone of different stages, we create a representation space suitable for multiple domains incrementally. The merged representation serves as a balanced intermediary that captures task-specific features from all seen domains. Additionally, to address the mismatch between consolidated embeddings and the classifier, we introduce an extra classifier consolidation process. Leveraging class-wise semantic information, we estimate the classifier weights of old domains within the latest embedding space. By merging historical and estimated classifiers, we align them with the consolidated embedding space, facilitating incremental classification. Extensive experimental results on four benchmark datasets demonstrate Duct's state-of-the-art performance. Code is available at https://github.com/Estrella-fugaz/CVPR25-Duct