Data to 7 Oct 2026

Lifelong Learning with Dynamically Expandable Networks

arXiv 1708.01547 ↗First posted 4 Aug 2017cs.LGcited by 689

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets under lifelong learning scenarios, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch counterparts with substantially fewer number of parameters. Further, the obtained network fine-tuned on all tasks obtained significantly better performance over the batch models, which shows that it can be used to estimate the optimal network structure even when all tasks are available in the first place.

Key facts. Cited by 689 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 107; 2022: 117; 2023: 140; 2024: 158; 2025: 105; 2026: 62.

689 papers.

Sort:

By area:

2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
2021Learning to Prompt for Continual Learningcited by 938
2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
2021Gradient Projection Memory for Continual Learningcited by 338
2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
2023Class-Incremental Learning: A Surveycited by 209
2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199

38 more

2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
2022Patching open-vocabulary models by interpolating weightscited by 187
2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
2021Towards Continual Knowledge Learning of Language Modelscited by 172
2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
2022Continual Learning and Private Unlearningcited by 124
2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 114
2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
2021Representational Continuity for Unsupervised Continual Learningcited by 95
2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 87
2021Invariant Information Bottleneck for Domain Generalizationcited by 86
2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
2024Continual Learning and Catastrophic Forgettingcited by 73
2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 71
2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 66
2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
2021Wide Neural Networks Forget Less Catastrophicallycited by 64
2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 64
2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
2021Continual Learning via Local Module Compositioncited by 61
2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61

50 most cited of 689 All 689 (JSON)