# Failure Modes of Always-On Inter-Cluster Repulsion in Replay-Based Continual Learning

arXiv: https://arxiv.org/abs/2510.07648 · HTML: https://mlsanity.com/paper/2510.07648/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 9 of 9)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [Preserving Linear Separability in Continual Learning by Backward Feature Projection](https://mlsanity.com/paper/2303.14595/) (2023)
- [Centroid Distance Distillation for Effective Rehearsal in Continual Learning](https://mlsanity.com/paper/2303.02954/) (2023)

## Abstract (arXiv)

Feature-space objectives are often added to replay-based continual learning systems with the expectation that better geometric separation will improve retention. We study a preliminary form of Cluster-Aware Replay (CAR) that combines a class-balanced replay memory with an always-active inter-cluster repulsion term (ICF). On five-task Split CIFAR-10 with a ResNet-18 backbone, the highest observed mean in a six-value sensitivity sweep reaches 22.5±1.4% final average accuracy over three seeds, compared with 23.1±2.5% for replay alone. ICF without replay reaches only 19.2±0.1%. All tested repulsion weights produce final accuracies between 20.1% and 22.5%, and the detailed configuration exhibits 89.2±1.5 percentage points of average forgetting. An instrumented rerun shows that the weighted repulsion contribution remains near -0.13 after cross-entropy has fallen close to zero, so the total objective becomes negative while old-task accuracy collapses. Importantly, the normalized distance objective is mathematically bounded; the failure is therefore better described as non-saturating, always-on repulsion rather than an unbounded loss. These negative results show that geometric separation is not automatically complementary to replay and motivate margin-gated objectives whose gradients deactivate once sufficient separation has been reached.