# RTRA: Rapid Training of Regularization-based Approaches in Continual Learning

arXiv: https://arxiv.org/abs/2312.09361 · HTML: https://mlsanity.com/paper/2312.09361/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [FoodX-251: A Dataset for Fine-grained Food Classification](https://mlsanity.com/paper/1907.06167/) (2019)
- [Less-forgetful Learning for Domain Expansion in Deep Neural Networks](https://mlsanity.com/paper/1711.05959/) (2017)
- [Accelerating Natural Gradient with Higher-Order Invariance](https://mlsanity.com/paper/1803.01273/) (2018)
- [EmbAu: A Novel Technique to Embed Audio Data Using Shuffled Frog Leaping Algorithm](https://mlsanity.com/paper/2312.08417/) (2023)

## Built on by (most cited first, 5 of 5)

- [Never-Ending Behavior-Cloning Agent for Robotic Manipulation](https://mlsanity.com/paper/2403.00336/) (2024)
- [On Understanding of the Dynamics of Model Capacity in Continual Learning](https://mlsanity.com/paper/2508.08052/) (2025)
- [Secure Information Embedding in Images with Hybrid Firefly Algorithm](https://mlsanity.com/paper/2312.13519/) (2023)
- [CLeAN: Continual Learning Adaptive Normalization in Dynamic Environments](https://mlsanity.com/paper/2603.17548/) (2026)
- [Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models](https://mlsanity.com/paper/2605.07512/) (2026)

## Abstract (arXiv)

Catastrophic forgetting(CF) is a significant challenge in continual learning (CL). In regularization-based approaches to mitigate CF, modifications to important training parameters are penalized in subsequent tasks using an appropriate loss function. We propose the RTRA, a modification to the widely used Elastic Weight Consolidation (EWC) regularization scheme, using the Natural Gradient for loss function optimization. Our approach improves the training of regularization-based methods without sacrificing test-data performance. We compare the proposed RTRA approach against EWC using the iFood251 dataset. We show that RTRA has a clear edge over the state-of-the-art approaches.