# Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones

arXiv: https://arxiv.org/abs/2205.01586 · HTML: https://mlsanity.com/paper/2205.01586/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Vision Transformers are Robust Learners](https://mlsanity.com/paper/2105.07581/) (2021)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis](https://mlsanity.com/paper/1909.01520/) (2019)
- [A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning](https://mlsanity.com/paper/2009.01797/) (2020)
- [Continuum: Simple Management of Complex Continual Learning Scenarios](https://mlsanity.com/paper/2102.06253/) (2021)
- [CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions](https://mlsanity.com/paper/2009.09929/) (2020)
- [Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization](https://mlsanity.com/paper/2203.13167/) (2022)
- [Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners](https://mlsanity.com/paper/2201.04924/) (2022)
- [Improving Vision Transformers for Incremental Learning](https://mlsanity.com/paper/2112.06103/) (2021)
- [Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers](https://mlsanity.com/paper/2106.13122/) (2021)
- [OvA-INN: Continual Learning with Invertible Neural Networks](https://mlsanity.com/paper/2006.13772/) (2020)

## Built on by (most cited first, 12 of 12)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [RanPAC: Random Projections and Pre-trained Models for Continual Learning](https://mlsanity.com/paper/2307.02251/) (2023)
- [Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning](https://mlsanity.com/paper/2403.18886/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [An Analysis of Initial Training Strategies for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2308.11677/) (2023)
- [Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning](https://mlsanity.com/paper/2403.07356/) (2024)
- [Chunking: Continual Learning is not just about Distribution Shift](https://mlsanity.com/paper/2310.02206/) (2023)
- [Sparse Orthogonal Parameters Tuning for Continual Learning](https://mlsanity.com/paper/2411.02813/) (2024)
- [CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation](https://mlsanity.com/paper/2507.19887/) (2025)
- [The Unreasonable Effectiveness of Randomized Representations in Online Continual Graph Learning](https://mlsanity.com/paper/2510.06819/) (2025)
- [CaMiT: A Time-Aware Car Model Dataset for Classification and Generation](https://mlsanity.com/paper/2510.17626/) (2025)
- [SAILS: Segment Anything with Incrementally Learned Semantics for Task-Invariant and Training-Free Continual Learning](https://mlsanity.com/paper/2602.14767/) (2026)

## Abstract (arXiv)

In this short paper, we propose a baseline (off-the-shelf) for Continual Learning of Computer Vision problems, by leveraging the power of pretrained models. By doing so, we devise a simple approach achieving strong performance for most of the common benchmarks. Our approach is fast since requires no parameters updates and has minimal memory requirements (order of KBytes). In particular, the "training" phase reorders data and exploit the power of pretrained models to compute a class prototype and fill a memory bank. At inference time we match the closest prototype through a knn-like approach, providing us the prediction. We will see how this naive solution can act as an off-the-shelf continual learning system. In order to better consolidate our results, we compare the devised pipeline with common CNN models and show the superiority of Vision Transformers, suggesting that such architectures have the ability to produce features of higher quality. Moreover, this simple pipeline, raises the same questions raised by previous works citegdumb on the effective progresses made by the CL community especially in the dataset considered and the usage of pretrained models. Code is live at https://github.com/francesco-p/off-the-shelf-cl