# On Understanding of the Dynamics of Model Capacity in Continual Learning

arXiv: https://arxiv.org/abs/2508.08052 · HTML: https://mlsanity.com/paper/2508.08052/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 30 of 35)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [Avalanche: an End-to-End Library for Continual Learning](https://mlsanity.com/paper/2104.00405/) (2021)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [Learning to Continually Learn](https://mlsanity.com/paper/2002.09571/) (2020)
- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [Maintaining Plasticity in Continual Learning via Regenerative Regularization](https://mlsanity.com/paper/2308.11958/) (2023)
- [Wide Neural Networks Forget Less Catastrophically](https://mlsanity.com/paper/2110.11526/) (2021)
- [Continual evaluation for lifelong learning: Identifying the stability gap](https://mlsanity.com/paper/2205.13452/) (2022)
- [Continual Learning in Linear Classification on Separable Data](https://mlsanity.com/paper/2306.03534/) (2023)
- [Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime](https://mlsanity.com/paper/2207.06475/) (2022)
- [Learnability and Algorithm for Continual Learning](https://mlsanity.com/paper/2306.12646/) (2023)
- [The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model](https://mlsanity.com/paper/2401.12617/) (2024)
- [Formalizing the Generalization-Forgetting Trade-off in Continual Learning](https://mlsanity.com/paper/2109.14035/) (2021)
- [Open-World Continual Learning: Unifying Novelty Detection and Continual Learning](https://mlsanity.com/paper/2304.10038/) (2023)
- [Investigating Forgetting in Pre-Trained Representations Through Continual Learning](https://mlsanity.com/paper/2305.05968/) (2023)
- [Improving Plasticity in Online Continual Learning via Collaborative Learning](https://mlsanity.com/paper/2312.00600/) (2023)
- [GRASP: A Rehearsal Policy for Efficient Online Continual Learning](https://mlsanity.com/paper/2308.13646/) (2023)
- [Overcoming the Stability Gap in Continual Learning](https://mlsanity.com/paper/2306.01904/) (2023)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning](https://mlsanity.com/paper/2312.16409/) (2023)
- [Continual Learning with Dirichlet Generative-based Rehearsal](https://mlsanity.com/paper/2309.06917/) (2023)
- [Adaptive Reorganization of Neural Pathways for Continual Learning with Spiking Neural Networks](https://mlsanity.com/paper/2309.09550/) (2023)
- [Adversarial Robust Memory-Based Continual Learner](https://mlsanity.com/paper/2311.17608/) (2023)

## Built on by (most cited first, 1 of 1)

- [What Does Model Growth Really Add? Functional Capacity Beyond Parameter Count](https://mlsanity.com/paper/2607.14571/) (2026)

## Abstract (arXiv)

The stability-plasticity dilemma, closely related to a neural network's (NN) capacity-its ability to represent tasks-is a fundamental challenge in continual learning (CL). Within this context, we introduce CL's effective model capacity (CLEMC) that characterizes the dynamic behavior of the stability-plasticity balance point. We develop a difference equation to model the evolution of the interplay between the NN, task data, and optimization procedure. We then leverage CLEMC to demonstrate that the effective capacity-and, by extension, the stability-plasticity balance point is inherently non-stationary. We show that regardless of the NN architecture or optimization method, a NN's ability to represent new tasks diminishes when incoming task distributions differ from previous ones. We conduct extensive experiments to support our theoretical findings, spanning a range of architectures-from small feedforward network and convolutional networks to medium-sized graph neural networks and transformer-based large language models with millions of parameters.