# Dense Network Expansion for Class Incremental Learning

arXiv: https://arxiv.org/abs/2303.12696 · HTML: https://mlsanity.com/paper/2303.12696/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022)
- [Encoder Based Lifelong Learning](https://mlsanity.com/paper/1704.01920/) (2017)
- [Continual Learning via Neural Pruning](https://mlsanity.com/paper/1903.04476/) (2019)
- [Class-Incremental Learning with Strong Pre-trained Models](https://mlsanity.com/paper/2204.03634/) (2022)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)

## Built on by (most cited first, 30 of 45)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters](https://mlsanity.com/paper/2403.11549/) (2024)
- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Make Continual Learning Stronger via C-Flat](https://mlsanity.com/paper/2404.00986/) (2024)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning](https://mlsanity.com/paper/2508.08165/) (2025)
- [LLaVA-c: Continual Improved Visual Instruction Tuning](https://mlsanity.com/paper/2506.08666/) (2025)
- [Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2509.16738/) (2025)
- [Learning More Generalized Experts by Merging Experts in Mixture-of-Experts](https://mlsanity.com/paper/2405.11530/) (2024)
- [LLMs Can Evolve Continually on Modality for X-Modal Reasoning](https://mlsanity.com/paper/2410.20178/) (2024)
- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [Versatile Incremental Learning: Towards Class and Domain-Agnostic Incremental Learning](https://mlsanity.com/paper/2409.10956/) (2024)
- [PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2407.14029/) (2024)
- [POP: Prompt Of Prompts for Continual Learning](https://mlsanity.com/paper/2306.08200/) (2023)
- [Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.14142/) (2024)
- [Non-exemplar Class-incremental Learning by Random Auxiliary Classes Augmentation and Mixed Features](https://mlsanity.com/paper/2304.07707/) (2023)
- [Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation](https://mlsanity.com/paper/2503.22136/) (2025)
- [CEAT: Continual Expansion and Absorption Transformer for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2403.06670/) (2024)
- [CSTA: Spatial-Temporal Causal Adaptive Learning for Exemplar-Free Video Class-Incremental Learning](https://mlsanity.com/paper/2501.07236/) (2025)
- [Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration](https://mlsanity.com/paper/2403.11373/) (2024)
- [Harnessing Neural Unit Dynamics for Effective and Scalable Class-Incremental Learning](https://mlsanity.com/paper/2406.02428/) (2024)
- [Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning](https://mlsanity.com/paper/2411.06764/) (2024)
- [C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models](https://mlsanity.com/paper/2502.17920/) (2025)
- [Energy-Structured Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2605.27482/) (2026)
- [LoRA-Loop: Closing the Synthetic Replay Cycle for Continual VLM Learning](https://mlsanity.com/paper/2507.13568/) (2025)
- [Exploiting Fine-Grained Prototype Distribution for Boosting Unsupervised Class Incremental Learning](https://mlsanity.com/paper/2408.10046/) (2024)
- [Continual Learning of Multiple Cognitive Functions with Brain-inspired Temporal Development Mechanism](https://mlsanity.com/paper/2504.05621/) (2025)
- [Proxy-Anchor and EVT-Driven Continual Learning Method for Generalized Category Discovery](https://mlsanity.com/paper/2504.08550/) (2025)
- [C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning](https://mlsanity.com/paper/2508.18860/) (2025)

## Abstract (arXiv)

The problem of class incremental learning (CIL) is considered. State-of-the-art approaches use a dynamic architecture based on network expansion (NE), in which a task expert is added per task. While effective from a computational standpoint, these methods lead to models that grow quickly with the number of tasks. A new NE method, dense network expansion (DNE), is proposed to achieve a better trade-off between accuracy and model complexity. This is accomplished by the introduction of dense connections between the intermediate layers of the task expert networks, that enable the transfer of knowledge from old to new tasks via feature sharing and reusing. This sharing is implemented with a cross-task attention mechanism, based on a new task attention block (TAB), that fuses information across tasks. Unlike traditional attention mechanisms, TAB operates at the level of the feature mixing and is decoupled with spatial attentions. This is shown more effective than a joint spatial-and-task attention for CIL. The proposed DNE approach can strictly maintain the feature space of old classes while growing the network and feature scale at a much slower rate than previous methods. In result, it outperforms the previous SOTA methods by a margin of 4% in terms of accuracy, with similar or even smaller model scale.