# Incremental Learning Through Deep Adaptation

arXiv: https://arxiv.org/abs/1705.04228 · HTML: https://mlsanity.com/paper/1705.04228/

## Key facts

- Cited by 108 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2017.

## Built on by (most cited first, 30 of 108)

- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [Continual Learning and Private Unlearning](https://mlsanity.com/paper/2203.12817/) (2022)
- [TRGP: Trust Region Gradient Projection for Continual Learning](https://mlsanity.com/paper/2202.02931/) (2022)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [Probing Representation Forgetting in Supervised and Unsupervised Continual Learning](https://mlsanity.com/paper/2203.13381/) (2022)
- [Self-Supervised Training Enhances Online Continual Learning](https://mlsanity.com/paper/2103.14010/) (2021)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [A Survey on Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2304.08130/) (2023)
- [Cluster-driven Graph Federated Learning over Multiple Domains](https://mlsanity.com/paper/2104.14628/) (2021)
- [Task Adaptive Parameter Sharing for Multi-Task Learning](https://mlsanity.com/paper/2203.16708/) (2022)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [Continually Learning Self-Supervised Representations with Projected Functional Regularization](https://mlsanity.com/paper/2112.15022/) (2021)
- [Continual Learning on Dynamic Graphs via Parameter Isolation](https://mlsanity.com/paper/2305.13825/) (2023)
- [Controllable Semantic Parsing via Retrieval Augmentation](https://mlsanity.com/paper/2110.08458/) (2021)
- [Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning](https://mlsanity.com/paper/2303.14771/) (2023)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [Unsupervised Visual Attention and Invariance for Reinforcement Learning](https://mlsanity.com/paper/2104.02921/) (2021)
- [Learning with Recoverable Forgetting](https://mlsanity.com/paper/2207.08224/) (2022)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)
- [ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systems](https://mlsanity.com/paper/2312.06573/) (2023)
- [LoTR: Low Tensor Rank Weight Adaptation](https://mlsanity.com/paper/2402.01376/) (2024)
- [Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation](https://mlsanity.com/paper/2211.06866/) (2022)
- [CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation](https://mlsanity.com/paper/2310.01407/) (2023)
- [Multi-Domain Incremental Learning for Semantic Segmentation](https://mlsanity.com/paper/2110.12205/) (2021)
- [Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches](https://mlsanity.com/paper/2203.14843/) (2022)
- [Universal Representations: A Unified Look at Multiple Task and Domain Learning](https://mlsanity.com/paper/2204.02744/) (2022)
- [Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey](https://mlsanity.com/paper/2412.13840/) (2024)
- [Improving Plasticity in Online Continual Learning via Collaborative Learning](https://mlsanity.com/paper/2312.00600/) (2023)
- [Collaborative Intelligence: Challenges and Opportunities](https://mlsanity.com/paper/2102.06841/) (2021)

## Abstract (arXiv)

Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added domain, typically as many as the original network. We propose a method called Deep Adaptation Networks (DAN) that constrains newly learned filters to be linear combinations of existing ones. DANs precisely preserve performance on the original domain, require a fraction (typically 13%, dependent on network architecture) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3% of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.