mlsanityData to 7 Oct 2026

Feature Forgetting in Continual Representation Learning

arXiv 2205.13359 ↗First posted 26 May 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In continual and lifelong learning, good representation learning can help increase performance and reduce sample complexity when learning new tasks. There is evidence that representations do not suffer from "catastrophic forgetting" even in plain continual learning, but little further fact is known about its characteristics. In this paper, we aim to gain more understanding about representation learning in continual learning, especially on the feature forgetting problem. We devise a protocol for evaluating representation in continual learning, and then use it to present an overview of the basic trends of continual representation learning, showing its consistent deficiency and potential issues. To study the feature forgetting problem, we create a synthetic dataset to identify and visualize the prevalence of feature forgetting in neural networks. Finally, we propose a simple technique using gating adapters to mitigate feature forgetting. We conclude by discussing that improving representation learning benefits both old and new tasks in continual learning.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

37 of 55 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2016Learning without Forgettingcited by 2,465
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  11. 2017Dynamic Routing Between Capsulescited by 689
  12. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
25 more
  1. 2019Three scenarios for continual learningcited by 610
  2. 2017Learning multiple visual domains with residual adapterscited by 553
  3. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  4. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  5. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  6. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  7. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  8. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  9. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  10. 2020Supermasks in Superpositioncited by 223
  11. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  12. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  13. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  14. 2018Towards Robust Evaluations of Continual Learningcited by 148
  15. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  16. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  17. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  18. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  19. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  20. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  21. 2017Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Executioncited by 59
  22. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  23. 2020Drinking from a Firehose: Continual Learning with Web-scale Natural Languagecited by 24
  24. 2020Measuring Information Transfer in Neural Networkscited by 4
  25. 2019Learning to Remember from a Multi-Task Teachercited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 2.

6 papers.

  1. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  2. 2023Preserving Linear Separability in Continual Learning by Backward Feature Projectioncited by 6
  3. 2024Gradual Divergence for Seamless Adaptation: A Novel Domain Incremental Learning Methodcited by 6
  4. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  5. 2025Heads collapse, features stay: Why Replay needs big bufferscited by 1
  6. 2024Examining Changes in Internal Representations of Continual Learning Models Through Tensor Decompositioncited by 0