# The Expanding Scope of the Stability Gap: Unveiling its Presence in Joint Incremental Learning of Homogeneous Tasks

arXiv: https://arxiv.org/abs/2406.05114 · HTML: https://mlsanity.com/paper/2406.05114/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [Linear Mode Connectivity in Multitask and Continual Learning](https://mlsanity.com/paper/2010.04495/) (2020)
- [Continual evaluation for lifelong learning: Identifying the stability gap](https://mlsanity.com/paper/2205.13452/) (2022)
- [Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also How](https://mlsanity.com/paper/2311.04898/) (2023)
- [Improving Online Continual Learning Performance and Stability with Temporal Ensembles](https://mlsanity.com/paper/2306.16817/) (2023)
- [Overcoming the Stability Gap in Continual Learning](https://mlsanity.com/paper/2306.01904/) (2023)

## Built on by (most cited first, 3 of 3)

- [Online Curvature-Aware Replay: Leveraging mathbf2nd Order Information for Online Continual Learning](https://mlsanity.com/paper/2502.01866/) (2025)
- [Exploring the Stability Gap in Continual Learning: The Role of the Classification Head](https://mlsanity.com/paper/2411.04723/) (2024)
- [Dynamic gain neuromodulation attenuates the stability gap under joint training](https://mlsanity.com/paper/2507.14056/) (2025)

## Abstract (arXiv)

Recent research identified a temporary performance drop on previously learned tasks when transitioning to a new one. This drop is called the stability gap and has great consequences for continual learning: it complicates the direct employment of continually learning since the worse-case performance at task-boundaries is dramatic, it limits its potential as an energy-efficient training paradigm, and finally, the stability drop could result in a reduced final performance of the algorithm. In this paper, we show that the stability gap also occurs when applying joint incremental training of homogeneous tasks. In this scenario, the learner continues training on the same data distribution and has access to all data from previous tasks. In addition, we show that in this scenario, there exists a low-loss linear path to the next minima, but that SGD optimization does not choose this path. We perform further analysis including a finer batch-wise analysis which could provide insights towards potential solution directions.