# Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractor

arXiv: https://arxiv.org/abs/2204.13349 · HTML: https://mlsanity.com/paper/2204.13349/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 78 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Compacting, Picking and Growing for Unforgetting Continual Learning](https://mlsanity.com/paper/1910.06562/) (2019)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [Adaptive Aggregation Networks for Class-Incremental Learning](https://mlsanity.com/paper/2010.05063/) (2020)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Uncertainty-based Continual Learning with Adaptive Regularization](https://mlsanity.com/paper/1905.11614/) (2019)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)

## Built on by (most cited first, 27 of 27)

- [Divide and not forget: Ensemble of selectively trained experts in Continual Learning](https://mlsanity.com/paper/2401.10191/) (2024)
- [Language Guided Concept Bottleneck Models for Interpretable Continual Learning](https://mlsanity.com/paper/2503.23283/) (2025)
- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [Generative Multi-modal Models are Good Class-Incremental Learners](https://mlsanity.com/paper/2403.18383/) (2024)
- [Adapter Learning in Pretrained Feature Extractor for Continual Learning of Diseases](https://mlsanity.com/paper/2304.09042/) (2023)
- [Handling Out-of-Distribution Data: A Survey](https://mlsanity.com/paper/2507.21160/) (2025)
- [Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2401.01598/) (2024)
- [SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models](https://mlsanity.com/paper/2411.06171/) (2024)
- [Where, What, Why: Towards Explainable Driver Attention Prediction](https://mlsanity.com/paper/2506.23088/) (2025)
- [Leveraging Old Knowledge to Continually Learn New Classes in Medical Images](https://mlsanity.com/paper/2303.13752/) (2023)
- [La-SoftMoE CLIP for Unified Physical-Digital Face Attack Detection](https://mlsanity.com/paper/2408.12793/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Continual Learning for Tumor Classification in Histopathology Images](https://mlsanity.com/paper/2208.03609/) (2022)
- [Data efficient deep learning for medical image analysis: A survey](https://mlsanity.com/paper/2310.06557/) (2023)
- [Continual Domain Incremental Learning for Privacy-aware Digital Pathology](https://mlsanity.com/paper/2409.06455/) (2024)
- [Class Impression for Data-free Incremental Learning](https://mlsanity.com/paper/2207.00005/) (2022)
- [Class Balance Matters to Active Class-Incremental Learning](https://mlsanity.com/paper/2412.06642/) (2024)
- [CCSI: Continual Class-Specific Impression for Data-free Class Incremental Learning](https://mlsanity.com/paper/2406.05631/) (2024)
- [Class Incremental Learning with Task-Specific Batch Normalization and Out-of-Distribution Detection](https://mlsanity.com/paper/2411.00430/) (2024)
- [YOLO-IOD: Towards Real Time Incremental Object Detection](https://mlsanity.com/paper/2512.22973/) (2025)
- [Foundation Models as Class-Incremental Learners for Dermatological Image Classification](https://mlsanity.com/paper/2507.14050/) (2025)
- [On Sequential Maximum a Posteriori Inference for Continual Learning](https://mlsanity.com/paper/2405.16498/) (2024)
- [MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning](https://mlsanity.com/paper/2502.02372/) (2025)
- [Cross-Domain Few-Shot Learning via Multi-View Collaborative Optimization with Vision-Language Models](https://mlsanity.com/paper/2508.12861/) (2025)
- [High Semantic Features for the Continual Learning of Complex Emotions: a Lightweight Solution](https://mlsanity.com/paper/2510.13534/) (2025)
- [Concept-wise Attention for Fine-grained Concept Bottleneck Models](https://mlsanity.com/paper/2604.15748/) (2026)
- [SPARS: A Stability-Plasticity-Aware Region Selection Framework for Continual Learning](https://mlsanity.com/paper/2610.19876/) (2026)

## Abstract (arXiv)

Deep learning has shown its human-level performance in various applications. However, current deep learning models are characterised by catastrophic forgetting of old knowledge when learning new classes. This poses a challenge particularly in intelligent diagnosis systems where initially only training data of a limited number of diseases are available. In this case, updating the intelligent system with data of new diseases would inevitably downgrade its performance on previously learned diseases. Inspired by the process of learning new knowledge in human brains, we propose a Bayesian generative model for continual learning built on a fixed pre-trained feature extractor. In this model, knowledge of each old class can be compactly represented by a collection of statistical distributions, e.g. with Gaussian mixture models, and naturally kept from forgetting in continual learning over time. Unlike existing class-incremental learning methods, the proposed approach is not sensitive to the continual learning process and can be additionally well applied to the data-incremental learning scenario. Experiments on multiple medical and natural image classification tasks showed that the proposed approach outperforms state-of-the-art approaches which even keep some images of old classes during continual learning of new classes.