# Context Channel Capacity: An Information-Theoretic Framework for Understanding Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2603.07415 · HTML: https://mlsanity.com/paper/2603.07415/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 8 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [Many Task Learning with Task Routing](https://mlsanity.com/paper/1903.12117/) (2019)

## Built on by (most cited first, 1 of 1)

- [The Sequential Price of Continual Learning](https://mlsanity.com/paper/2609.29674/) (2026)

## Abstract (arXiv)

Catastrophic forgetting remains a central challenge in continual learning (CL), yet lacks a unified information-theoretic explanation for why some architectures forget catastrophically while others do not. We introduce Context Channel Capacity (Cctx), the mutual information between a CL architecture's context signal and its generated parameters, and prove that zero forgetting requires Cctx ≥ H(T), where H(T) is the task identity entropy. We establish an Impossibility Triangle -- zero forgetting, online learning, and finite parameters cannot be simultaneously satisfied by sequential state-based learners -- and show that conditional regeneration architectures (HyperNetworks) bypass this triangle by redefining parameters as function values rather than states. We validate this framework across 8 CL methods on Split-MNIST (1,130+ experiments over 86 days, 4 seeds each), showing that Cctx perfectly predicts forgetting behavior: methods with Cctx = 0 (NaiveSGD, EWC, SI, LwF, CFlow) exhibit catastrophic forgetting (6--97%), while methods with Cctx ≈ 1 (HyperNetwork) achieve zero forgetting (98.8% ACC). We further propose Wrong-Context Probing (P5), a practical diagnostic protocol for measuring Cctx, and extend the framework to CIFAR-10 via a novel Gradient Context Encoder that closes the oracle gap from 23.3pp to 0.7pp. A systematic taxonomy of 15+ closed research directions -- including the Hebbian null result (frozen random features outperform learned features), CFlow's θ0-memorizer phenomenon, and the SN symmetry barrier to column specialization -- provides the community with precisely diagnosed negative results. Our central design principle: architecture over algorithm -- the context pathway must be structurally unbypassable.