mlsanityData to 7 Oct 2026

Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

arXiv 1805.07810 ↗First posted 20 May 2018stat.ML cs.LGcited by 197

Not on the map. Why?

Abstract (arXiv)

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90% test accuracy across a sequence of 50 instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 37; 2022: 34; 2023: 36; 2024: 37; 2025: 35; 2026: 18.

197 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  3. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  4. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  5. 2023Mitigating the Alignment Tax of RLHFcited by 181
  6. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  7. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
  8. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  9. 2022A Theoretical Study on Solving Continual Learningcited by 78
  10. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  11. 2024Continual Learning and Catastrophic Forgettingcited by 73
  12. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 71
38 more
  1. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
  2. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  3. 2024Active Preference Learning for Large Language Modelscited by 55
  4. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  5. 2021Self-Supervised Training Enhances Online Continual Learningcited by 53
  6. 2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
  7. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  8. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  9. 2022Online Continual Learning for Embedded Devicescited by 48
  10. 2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
  11. 2021Posterior Meta-Replay for Continual Learningcited by 45
  12. 2023Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligencecited by 43
  13. 2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 43
  14. 2021A Bayesian Federated Learning Framework with Online Laplace Approximationcited by 41
  15. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  16. 2021DualNet: Continual Learning, Fast and Slowcited by 39
  17. 2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 39
  18. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  19. 2021Understanding Continual Learning Settings with Data Distribution Drift Analysiscited by 38
  20. 2024Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learningcited by 37
  21. 2021Non-IID data and Continual Learning processes in Federated Learning: A long road aheadcited by 36
  22. 2021Curriculum-Meta Learning for Order-Robust Continual Relation Extractioncited by 34
  23. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  24. 2022Continual Learning with Recursive Gradient Optimizationcited by 32
  25. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  26. 2025Activation-Informed Merging of Large Language Modelscited by 29
  27. 2023Learnability and Algorithm for Continual Learningcited by 28
  28. 2023Continual Learning via Sequential Function-Space Variational Inferencecited by 28
  29. 2021Kernel Continual Learningcited by 27
  30. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  31. 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27
  32. 2023Continual Learning with Low Rank Adaptationcited by 27
  33. 2023Tangent Model Composition for Ensembling and Continual Fine-tuningcited by 25
  34. 2021A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applicationscited by 24
  35. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  36. 2023SIESTA: Efficient Online Continual Learning with Sleepcited by 24
  37. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  38. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24

50 most cited of 197 All 197 (JSON)