Failure Modes of Always-On Inter-Cluster Repulsion in Replay-Based Continual Learning

Feature-space objectives are often added to replay-based continual learning systems with the expectation that better geometric separation will improve retention. We study a preliminary form of Cluster-Aware Replay (CAR) that combines a class-balanced replay memory with an always-active inter-cluster repulsion term (ICF). On five-task Split CIFAR-10 with a ResNet-18 backbone, the highest observed mean in a six-value sensitivity sweep reaches 22.5±1.4% final average accuracy over three seeds, compared with 23.1±2.5% for replay alone. ICF without replay reaches only 19.2±0.1%. All tested repulsion weights produce final accuracies between 20.1% and 22.5%, and the detailed configuration exhibits 89.2±1.5 percentage points of average forgetting. An instrumented rerun shows that the weighted repulsion contribution remains near -0.13 after cross-entropy has fallen close to zero, so the total objective becomes negative while old-task accuracy collapses. Importantly, the normalized distance objective is mathematically bounded; the failure is therefore better described as non-saturating, always-on repulsion rather than an unbounded loss. These negative results show that geometric separation is not automatically complementary to replay and motivate margin-gated objectives whose gradients deactivate once sufficient separation has been reached.