# Continual Learning with Extended Kronecker-factored Approximate Curvature

arXiv: https://arxiv.org/abs/2004.07507 · HTML: https://mlsanity.com/paper/2004.07507/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2020.

## Built on by (most cited first, 30 of 39)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping](https://mlsanity.com/paper/2102.11343/) (2021)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)
- [Tangent Model Composition for Ensembling and Continual Fine-tuning](https://mlsanity.com/paper/2307.08114/) (2023)
- [Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning](https://mlsanity.com/paper/2508.08165/) (2025)
- [Revisiting Neural Networks for Continual Learning: An Architectural Perspective](https://mlsanity.com/paper/2404.14829/) (2024)
- [Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learning](https://mlsanity.com/paper/2201.12559/) (2022)
- [Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting](https://mlsanity.com/paper/2502.02797/) (2025)
- [DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning](https://mlsanity.com/paper/2208.08112/) (2022)
- [Continual Learning for Predictive Maintenance: Overview and Challenges](https://mlsanity.com/paper/2301.12467/) (2023)
- [Coarse-To-Fine Incremental Few-Shot Learning](https://mlsanity.com/paper/2111.14806/) (2021)
- [DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics](https://mlsanity.com/paper/2403.14353/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data](https://mlsanity.com/paper/2210.05508/) (2022)
- [Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks](https://mlsanity.com/paper/2107.06533/) (2021)
- [Continual Hyperbolic Learning of Instances and Classes](https://mlsanity.com/paper/2506.10710/) (2025)
- [Exemplar-free Continual Learning of Vision Transformers via Gated Class-Attention and Cascaded Feature Drift Compensation](https://mlsanity.com/paper/2211.12292/) (2022)
- [Integral Continual Learning Along the Tangent Vector Field of Tasks](https://mlsanity.com/paper/2211.13108/) (2022)
- [Remind of the Past: Incremental Learning with Analogical Prompts](https://mlsanity.com/paper/2303.13898/) (2023)
- [Self-Regulated Neurogenesis for Online Data-Incremental Learning](https://mlsanity.com/paper/2403.14684/) (2024)
- [Class-Incremental Learning Using Generative Experience Replay Based on Time-aware Regularization](https://mlsanity.com/paper/2310.03898/) (2023)
- [Enhancing Plasticity for First Session Adaptation Continual Learning](https://mlsanity.com/paper/2310.11482/) (2023)
- [Constructing Enhanced Mutual Information for Online Class-Incremental Learning](https://mlsanity.com/paper/2407.18526/) (2024)
- [Flashbacks to Harmonize Stability and Plasticity in Continual Learning](https://mlsanity.com/paper/2506.00477/) (2025)
- [Densely Distilling Cumulative Knowledge for Continual Learning](https://mlsanity.com/paper/2405.09820/) (2024)
- [Continual Learning With Quasi-Newton Methods](https://mlsanity.com/paper/2503.19939/) (2025)
- [Learning Yourself: Class-Incremental Semantic Segmentation with Language-Inspired Bootstrapped Disentanglement](https://mlsanity.com/paper/2509.00527/) (2025)

## Abstract (arXiv)

We propose a quadratic penalty method for continual learning of neural networks that contain batch normalization (BN) layers. The Hessian of a loss function represents the curvature of the quadratic penalty function, and a Kronecker-factored approximate curvature (K-FAC) is used widely to practically compute the Hessian of a neural network. However, the approximation is not valid if there is dependence between examples, typically caused by BN layers in deep network architectures. We extend the K-FAC method so that the inter-example relations are taken into account and the Hessian of deep neural networks can be properly approximated under practical assumptions. We also propose a method of weight merging and reparameterization to properly handle statistical parameters of BN, which plays a critical role for continual learning with BN, and a method that selects hyperparameters without source task data. Our method shows better performance than baselines in the permuted MNIST task with BN layers and in sequential learning from the ImageNet classification task to fine-grained classification tasks with ResNet-50, without any explicit or implicit use of source task data for hyperparameter selection.