# On Quadratic Penalties in Elastic Weight Consolidation

arXiv: https://arxiv.org/abs/1712.03847 · HTML: https://mlsanity.com/paper/1712.03847/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2017.

## Built on by (most cited first, 30 of 55)

- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Model Merging by Uncertainty-Based Gradient Matching](https://mlsanity.com/paper/2310.12808/) (2023)
- [Posterior Meta-Replay for Continual Learning](https://mlsanity.com/paper/2103.01133/) (2021)
- [Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement](https://mlsanity.com/paper/2409.19732/) (2024)
- [DualNet: Continual Learning, Fast and Slow](https://mlsanity.com/paper/2110.00175/) (2021)
- [Open-world Machine Learning: Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2105.13448/) (2021)
- [Neuro-Symbolic Continual Learning: Knowledge, Reasoning Shortcuts and Concept Rehearsal](https://mlsanity.com/paper/2302.01242/) (2023)
- [A Statistical Theory of Regularization-Based Continual Learning](https://mlsanity.com/paper/2406.06213/) (2024)
- [CLeaR: An Adaptive Continual Learning Framework for Regression Tasks](https://mlsanity.com/paper/2101.00926/) (2021)
- [Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis](https://mlsanity.com/paper/2104.09396/) (2021)
- [How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation](https://mlsanity.com/paper/2102.02805/) (2021)
- [A Second-Order Perspective on Model Compositionality and Incremental Learning](https://mlsanity.com/paper/2405.16350/) (2024)
- [Scalable Bayesian Learning with posteriors](https://mlsanity.com/paper/2406.00104/) (2024)
- [On the Computation of the Fisher Information in Continual Learning](https://mlsanity.com/paper/2502.11756/) (2025)
- [EVCL: Elastic Variational Continual Learning with Weight Consolidation](https://mlsanity.com/paper/2406.15972/) (2024)
- [BECAME: BayEsian Continual Learning with Adaptive Model MErging](https://mlsanity.com/paper/2504.02666/) (2025)
- [Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning](https://mlsanity.com/paper/2109.10021/) (2021)
- [Deep Reinforcement Learning for Autonomous Cyber Defence: A Survey](https://mlsanity.com/paper/2310.07745/) (2023)
- [Knowledge Retention for Continual Model-Based Reinforcement Learning](https://mlsanity.com/paper/2503.04256/) (2025)
- [Towards continual task learning in artificial neural networks: current approaches and insights from neuroscience](https://mlsanity.com/paper/2112.14146/) (2021)
- [A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning](https://mlsanity.com/paper/2505.15178/) (2025)
- [A Multi-Memory Segment System for Generating High-Quality Long-Term Memory Content in Agents](https://mlsanity.com/paper/2508.15294/) (2025)
- [Approximate Bayesian Class-Conditional Models under Continuous Representation Shift](https://mlsanity.com/paper/2305.19076/) (2023)
- [CORE: Mitigating Catastrophic Forgetting in Continual Learning through Cognitive Replay](https://mlsanity.com/paper/2402.01348/) (2024)
- [Region-Based Optimization in Continual Learning for Audio Deepfake Detection](https://mlsanity.com/paper/2412.11551/) (2024)
- [Pareto Continual Learning: Preference-Conditioned Learning and Adaption for Dynamic Stability-Plasticity Trade-off](https://mlsanity.com/paper/2503.23390/) (2025)
- [Continuous Evolution Pool: Taming Recurring Concept Drift in Online Time Series Forecasting](https://mlsanity.com/paper/2506.14790/) (2025)
- [Revisiting Weight Regularization for Low-Rank Continual Learning](https://mlsanity.com/paper/2602.17559/) (2026)
- [Information-Theoretic Constraints for Continual Vision-Language-Action Alignment](https://mlsanity.com/paper/2603.13335/) (2026)

## Abstract (arXiv)

Elastic weight consolidation (EWC, Kirkpatrick et al, 2017) is a novel algorithm designed to safeguard against catastrophic forgetting in neural networks. EWC can be seen as an approximation to Laplace propagation (Eskin et al, 2004), and this view is consistent with the motivation given by Kirkpatrick et al (2017). In this note, I present an extended derivation that covers the case when there are more than two tasks. I show that the quadratic penalties in EWC are inconsistent with this derivation and might lead to double-counting data from earlier tasks.