mlsanityData to 7 Oct 2026

Task-agnostic Continual Learning with Hybrid Probabilistic Models

arXiv 2106.12772 ↗First posted 24 Jun 2021cs.LG stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Learning new tasks continuously without forgetting on a constantly changing data distribution is essential for real-world problems but extremely challenging for modern deep learning. In this work we propose HCL, a Hybrid generative-discriminative approach to Continual Learning for classification. We model the distribution of each task and each class with a normalizing flow. The flow is used to learn the data distribution, perform classification, identify task changes, and avoid forgetting, all leveraging the invertibility and exact likelihood which are uniquely enabled by the normalizing flow model. We use the generative capabilities of the flow to avoid catastrophic forgetting through generative replay and a novel functional regularization technique. For task identification, we use state-of-the-art anomaly detection techniques based on measuring the typicality of the model's statistics. We demonstrate the strong performance of HCL on a range of continual learning benchmarks such as split-MNIST, split-CIFAR, and SVHN-MNIST.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 57 of its 79 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

57 of 79 references.

References · 57

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016Layer Normalizationcited by 5,579
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2016Density estimation using Real NVPcited by 2,029
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  10. 2016Progressive Neural Networkscited by 1,332
  11. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  12. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
38 more
  1. 2017Continual Learning with Deep Generative Replaycited by 1,152
  2. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  3. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  4. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  5. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  6. 2019Three scenarios for continual learningcited by 610
  7. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  8. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  9. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  10. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  11. 2018Invertible Residual Networkscited by 364
  12. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  13. 2021Gradient Projection Memory for Continual Learningcited by 338
  14. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  15. 2015Highway Networkscited by 257
  16. 2019Residual Flows for Invertible Generative Modelingcited by 256
  17. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  18. 2020Supermasks in Superpositioncited by 223
  19. 2020Why Normalizing Flows Fail to Detect Out-of-Distribution Datacited by 220
  20. 2018Task-Free Continual Learningcited by 219
  21. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  22. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 160
  23. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  24. 2019Continual Unsupervised Representation Learningcited by 147
  25. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  26. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 133
  27. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 121
  28. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  29. 2020Hybrid Models for Open Set Recognitioncited by 109
  30. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 104
  31. 2019Semi-Supervised Learning with Normalizing Flowscited by 76
  32. 2020Density of States Estimation for Out-of-Distribution Detectioncited by 74
  33. 2018Task Agnostic Continual Learning Using Online Variational Bayescited by 61
  34. 2019Task Agnostic Continual Learning via Meta Learningcited by 53
  35. 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
  36. 2019Hybrid Models with Deep and Invertible Featurescited by 41
  37. 2020Learning latent representations across multiple data domains using Lifelong VAEGANcited by 35
  38. 2019Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognitioncited by 26

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 1; 2024: 4; 2025: 2; 2026: 1.

16 papers.

Sort:

By area:

  1. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  2. 2022Architecture Matters in Continual Learningcited by 46
  3. 2022Continual Learning Beyond a Single Modelcited by 18
  4. 2021One Pass ImageNetcited by 11
  5. 2022Continual Learning Approaches for Anomaly Detectioncited by 9
  6. 2022TAME: Task Agnostic Continual Learning using Multiple Expertscited by 7
  7. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  8. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 6
  9. 2022Continual learning on 3D point clouds with random compressed rehearsalcited by 5
  10. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 5
  11. 2024Collaborative Knowledge Distillation via a Learning-by-Education Node Communitycited by 1
  12. 2025Frugal Incremental Generative Modeling using Variational Autoencoderscited by 1
4 more
  1. 2022GMM-IL: Image Classification using Incrementally Learnt, Independent Probabilistic Models for Small Sample Sizescited by 0
  2. 2024Retrospective Feature Estimation for Continual Learningcited by 0
  3. 2024Hierarchically Gated Experts for Efficient Online Continual Learningcited by 0
  4. 2025Deep Generative Continual Learning using Functional LoRA: FunLoRAcited by 0