# Task-agnostic Continual Learning with Hybrid Probabilistic Models

arXiv: https://arxiv.org/abs/2106.12772 · HTML: https://mlsanity.com/paper/2106.12772/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 79 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 57)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Residual Flows for Invertible Generative Modeling](https://mlsanity.com/paper/1906.02735/) (2019)
- [Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines](https://mlsanity.com/paper/1810.12488/) (2018)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)

## Built on by (most cited first, 16 of 16)

- [Wide Neural Networks Forget Less Catastrophically](https://mlsanity.com/paper/2110.11526/) (2021)
- [Architecture Matters in Continual Learning](https://mlsanity.com/paper/2202.00275/) (2022)
- [Continual Learning Beyond a Single Model](https://mlsanity.com/paper/2202.09826/) (2022)
- [One Pass ImageNet](https://mlsanity.com/paper/2111.01956/) (2021)
- [Continual Learning Approaches for Anomaly Detection](https://mlsanity.com/paper/2212.11192/) (2022)
- [TAME: Task Agnostic Continual Learning using Multiple Experts](https://mlsanity.com/paper/2210.03869/) (2022)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Efficient Parametric Approximations of Neural Network Function Space Distance](https://mlsanity.com/paper/2302.03519/) (2023)
- [Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay](https://mlsanity.com/paper/2605.26097/) (2026)
- [Continual learning on 3D point clouds with random compressed rehearsal](https://mlsanity.com/paper/2205.08013/) (2022)
- [Collaborative Knowledge Distillation via a Learning-by-Education Node Community](https://mlsanity.com/paper/2410.00074/) (2024)
- [Frugal Incremental Generative Modeling using Variational Autoencoders](https://mlsanity.com/paper/2505.22408/) (2025)
- [GMM-IL: Image Classification using Incrementally Learnt, Independent Probabilistic Models for Small Sample Sizes](https://mlsanity.com/paper/2212.00572/) (2022)
- [Retrospective Feature Estimation for Continual Learning](https://mlsanity.com/paper/2406.17381/) (2024)
- [Hierarchically Gated Experts for Efficient Online Continual Learning](https://mlsanity.com/paper/2412.17188/) (2024)
- [Deep Generative Continual Learning using Functional LoRA: FunLoRA](https://mlsanity.com/paper/2510.02631/) (2025)

## Abstract (arXiv)

Learning new tasks continuously without forgetting on a constantly changing data distribution is essential for real-world problems but extremely challenging for modern deep learning. In this work we propose HCL, a Hybrid generative-discriminative approach to Continual Learning for classification. We model the distribution of each task and each class with a normalizing flow. The flow is used to learn the data distribution, perform classification, identify task changes, and avoid forgetting, all leveraging the invertibility and exact likelihood which are uniquely enabled by the normalizing flow model. We use the generative capabilities of the flow to avoid catastrophic forgetting through generative replay and a novel functional regularization technique. For task identification, we use state-of-the-art anomaly detection techniques based on measuring the typicality of the model's statistics. We demonstrate the strong performance of HCL on a range of continual learning benchmarks such as split-MNIST, split-CIFAR, and SVHN-MNIST.