# Incremental Learning via Rate Reduction

arXiv: https://arxiv.org/abs/2011.14593 · HTML: https://mlsanity.com/paper/2011.14593/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 30 of 31)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [RMM: Reinforced Memory Management for Class-Incremental Learning](https://mlsanity.com/paper/2301.05792/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Co-Transport for Class-Incremental Learning](https://mlsanity.com/paper/2107.12654/) (2021)
- [Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation](https://mlsanity.com/paper/2411.05663/) (2024)
- [Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models](https://mlsanity.com/paper/2306.05272/) (2023)
- [Is forgetting less a good inductive bias for forward transfer?](https://mlsanity.com/paper/2303.08207/) (2023)
- [Towards Robust Graph Incremental Learning on Evolving Graphs](https://mlsanity.com/paper/2402.12987/) (2024)
- [Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction](https://mlsanity.com/paper/2111.06636/) (2021)
- [Incremental Learning of Structured Memory via Closed-Loop Transcription](https://mlsanity.com/paper/2202.05411/) (2022)
- [Heterogeneous Forgetting Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2308.03374/) (2023)
- [CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks](https://mlsanity.com/paper/2103.04032/) (2021)
- [Understanding Neural Networks with Logarithm Determinant Entropy Estimator](https://mlsanity.com/paper/2105.03705/) (2021)
- [Never-Ending Behavior-Cloning Agent for Robotic Manipulation](https://mlsanity.com/paper/2403.00336/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Confidence Self-Calibration for Multi-Label Class-Incremental Learning](https://mlsanity.com/paper/2403.12559/) (2024)
- [Lifelong Language-Conditioned Robotic Manipulation Learning](https://mlsanity.com/paper/2603.05160/) (2026)
- [Hierarchical Augmentation and Distillation for Class Incremental Audio-Visual Video Recognition](https://mlsanity.com/paper/2401.06287/) (2024)
- [Lifelong Embodied Navigation Learning](https://mlsanity.com/paper/2603.06073/) (2026)
- [Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning](https://mlsanity.com/paper/2401.06548/) (2024)
- [Rebalancing Multi-Label Class-Incremental Learning](https://mlsanity.com/paper/2408.12161/) (2024)
- [All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation](https://mlsanity.com/paper/2603.14276/) (2026)
- [A classification performance evaluation measure considering data separability](https://mlsanity.com/paper/2211.05433/) (2022)
- [SCORE: Soft Label Compression-Centric Dataset Condensation via Coding Rate Optimization](https://mlsanity.com/paper/2503.13935/) (2025)
- [Proxy-Anchor and EVT-Driven Continual Learning Method for Generalized Category Discovery](https://mlsanity.com/paper/2504.08550/) (2025)
- [Lifelong Learning with Behavior Consolidation for Vehicle Routing](https://mlsanity.com/paper/2509.21765/) (2025)
- [Task-conditioned Ensemble of Expert Models for Continuous Learning](https://mlsanity.com/paper/2504.08626/) (2025)
- [Multi-Scale Structural Features for Continual, Comprehensible Visual Recognition in a Developmental Learning Framework](https://mlsanity.com/paper/2607.25531/) (2026)

## Abstract (arXiv)

Current deep learning architectures suffer from catastrophic forgetting, a failure to retain knowledge of previously learned classes when incrementally trained on new classes. The fundamental roadblock faced by deep learning methods is that deep learning models are optimized as "black boxes," making it difficult to properly adjust the model parameters to preserve knowledge about previously seen data. To overcome the problem of catastrophic forgetting, we propose utilizing an alternative "white box" architecture derived from the principle of rate reduction, where each layer of the network is explicitly computed without back propagation. Under this paradigm, we demonstrate that, given a pre-trained network and new data classes, our approach can provably construct a new network that emulates joint training with all past and new classes. Finally, our experiments show that our proposed learning algorithm observes significantly less decay in classification performance, outperforming state of the art methods on MNIST and CIFAR-10 by a large margin and justifying the use of "white box" algorithms for incremental learning even for sufficiently complex image data.