# Compact Memory for Continual Logistic Regression

arXiv: https://arxiv.org/abs/2511.09167 · HTML: https://mlsanity.com/paper/2511.09167/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Functional Regularisation for Continual Learning with Gaussian Processes](https://mlsanity.com/paper/1901.11356/) (2019)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [Continual Deep Learning by Functional Regularisation of Memorable Past](https://mlsanity.com/paper/2004.14070/) (2020)
- [Uncertainty-guided Continual Learning with Bayesian Neural Networks](https://mlsanity.com/paper/1906.02425/) (2019)
- [Information-theoretic Online Memory Selection for Continual Learning](https://mlsanity.com/paper/2204.04763/) (2022)
- [Avalanche: A PyTorch Library for Deep Continual Learning](https://mlsanity.com/paper/2302.01766/) (2023)
- [Understanding Forgetting in Continual Learning with Linear Regression](https://mlsanity.com/paper/2405.17583/) (2024)
- [Fixed Design Analysis of Regularization-Based Continual Learning](https://mlsanity.com/paper/2303.10263/) (2023)
- [Knowledge-Adaptation Priors](https://mlsanity.com/paper/2106.08769/) (2021)
- [Function-space Parameterization of Neural Networks for Sequential Learning](https://mlsanity.com/paper/2403.10929/) (2024)
- [Memory-Statistics Tradeoff in Continual Learning with Structural Regularization](https://mlsanity.com/paper/2504.04039/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Sequential Price of Continual Learning](https://mlsanity.com/paper/2609.29674/) (2026)

## Abstract (arXiv)

Despite recent progress, continual learning still does not match the performance of batch training. To avoid catastrophic forgetting, we need to build compact memory of essential past knowledge, but no clear solution has yet emerged, even for shallow neural networks with just one or two layers. In this paper, we present a new method to build compact memory for logistic regression. Our method is based on a result by Khan and Swaroop [2021] who show the existence of optimal memory for such models. We formulate the search for the optimal memory as Hessian-matching and propose a probabilistic PCA method to estimate them. Our approach can drastically improve accuracy compared to Experience Replay. For instance, on Split-ImageNet, we get 60% accuracy compared to 30% obtained by replay with memory-size equivalent to 0.3% of the data size. Increasing the memory size to 2% further boosts the accuracy to 74%, closing the gap to the batch accuracy of 77.6% on this task. Our work opens a new direction for building compact memory that can also be useful in the future for continual deep learning.