# Continual Feature Selection: Spurious Features in Continual Learning

arXiv: https://arxiv.org/abs/2203.01012 · HTML: https://mlsanity.com/paper/2203.01012/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Gradient Starvation: A Learning Proclivity in Neural Networks](https://mlsanity.com/paper/2011.09468/) (2020)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)
- [La-MAML: Look-ahead Meta Learning for Continual Learning](https://mlsanity.com/paper/2007.13904/) (2020)
- [Fortuitous Forgetting in Connectionist Networks](https://mlsanity.com/paper/2202.00155/) (2022)
- [Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning](https://mlsanity.com/paper/2003.05856/) (2020)
- [Understanding Continual Learning Settings with Data Distribution Drift Analysis](https://mlsanity.com/paper/2104.01678/) (2021)
- [Continuum: Simple Management of Complex Continual Learning Scenarios](https://mlsanity.com/paper/2102.06253/) (2021)
- [Learning where to learn: Gradient sparsity in meta and continual learning](https://mlsanity.com/paper/2110.14402/) (2021)
- [Regularization Shortcomings for Continual Learning](https://mlsanity.com/paper/1912.03049/) (2019)
- [Dropout as an Implicit Gating Mechanism For Continual Learning](https://mlsanity.com/paper/2004.11545/) (2020)
- [Learning Causal Models Online](https://mlsanity.com/paper/2006.07461/) (2020)
- [Continual Learning in Deep Networks: an Analysis of the Last Layer](https://mlsanity.com/paper/2106.01834/) (2021)

## Built on by (most cited first, 7 of 7)

- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning](https://mlsanity.com/paper/2304.05260/) (2023)
- [The Importance of Robust Features in Mitigating Catastrophic Forgetting](https://mlsanity.com/paper/2306.17091/) (2023)
- [Principles of Forgetting in Domain-Incremental Semantic Segmentation in Adverse Weather Conditions](https://mlsanity.com/paper/2303.14115/) (2023)
- [Watch Your Step: Optimal Retrieval for Continual Learning at Scale](https://mlsanity.com/paper/2404.10758/) (2024)
- [Continual Learning in the Presence of Spurious Correlation](https://mlsanity.com/paper/2303.11863/) (2023)
- [Selective Attention-based Modulation for Continual Learning](https://mlsanity.com/paper/2403.20086/) (2024)

## Abstract (arXiv)

Continual Learning (CL) is the research field addressing learning without forgetting when the data distribution is not static. This paper studies spurious features' influence on continual learning algorithms. We show that continual learning algorithms solve tasks by selecting features that are not generalizable. Our experiments highlight that continual learning algorithms face two related problems: (1) spurious features and (2) local spurious features. The first one is due to a covariate shift between training and testing data, while the second is due to the limited access to data at each training step. We study (1) through a consistent set of continual learning experiments varying spurious correlation amount and data distribution support. We show that (2) is a major cause of performance decrease in continual learning along with catastrophic forgetting. This paper presents a different way of understanding performance decrease in continual learning by highlighting the influence of (local) spurious features in algorithms capabilities.