# Degradation of Feature Space in Continual Learning

arXiv: https://arxiv.org/abs/2602.06586 · HTML: https://mlsanity.com/paper/2602.06586/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 13 of 13)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2309.14062/) (2023)
- [IsoScore: Measuring the Uniformity of Embedding Space Utilization](https://mlsanity.com/paper/2108.07344/) (2021)
- [Exploring Data Geometry for Continual Learning](https://mlsanity.com/paper/2304.03931/) (2023)
- [Stable Anisotropic Regularization](https://mlsanity.com/paper/2305.19358/) (2023)
- [Memory-efficient Continual Learning with Neural Collapse Contrastive](https://mlsanity.com/paper/2412.02865/) (2024)
- [Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay](https://mlsanity.com/paper/2404.01828/) (2024)

## Abstract (arXiv)

Centralized training is the standard paradigm in deep learning, enabling models to learn from a unified dataset in a single location. In such setup, isotropic feature distributions naturally arise as a mean to support well-structured and generalizable representations. In contrast, continual learning operates on streaming and non-stationary data, and trains models incrementally, inherently facing the well-known plasticity-stability dilemma. In such settings, learning dynamics tends to yield increasingly anisotropic feature space. This arises a fundamental question: should isotropy be enforced to achieve a better balance between stability and plasticity, and thereby mitigate catastrophic forgetting? In this paper, we investigate whether promoting feature-space isotropy can enhance representation quality in continual learning. Through experiments using contrastive continual learning techniques on CIFAR-10 and CIFAR-100 data, we find that isotropic regularization fails to improve, and can in fact degrade, model accuracy in continual settings. Our results highlight essential differences in feature geometry between centralized and continual learning, suggesting that isotropy, while beneficial in centralized setups, may not constitute an appropriate inductive bias for non-stationary learning scenarios.