# Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation

arXiv: https://arxiv.org/abs/1908.02984 · HTML: https://mlsanity.com/paper/1908.02984/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2019.

## Built on by (most cited first, 22 of 22)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [On the Stability-Plasticity Dilemma of Class-Incremental Learning](https://mlsanity.com/paper/2304.01663/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Layerwise Optimization by Gradient Decomposition for Continual Learning](https://mlsanity.com/paper/2105.07561/) (2021)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [Continual Learning with Weight Interpolation](https://mlsanity.com/paper/2404.04002/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Memory-efficient Continual Learning with Neural Collapse Contrastive](https://mlsanity.com/paper/2412.02865/) (2024)
- [Self-Activating Neural Ensembles for Continual Reinforcement Learning](https://mlsanity.com/paper/2301.00141/) (2022)
- [SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapse](https://mlsanity.com/paper/2510.15751/) (2025)
- [Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weights](https://mlsanity.com/paper/2604.17578/) (2026)
- [ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning](https://mlsanity.com/paper/2103.12216/) (2021)
- [SIRe-Networks: Convolutional Neural Networks Architectural Extension for Information Preservation via Skip/Residual Connections and Interlaced Auto-Encoders](https://mlsanity.com/paper/2110.02776/) (2021)
- [Evolving Knowledge Mining for Class Incremental Segmentation](https://mlsanity.com/paper/2306.02027/) (2023)
- [Evaluating and Improving Continual Learning in Spoken Language Understanding](https://mlsanity.com/paper/2402.10427/) (2024)
- [Low-Rank Filtering and Smoothing for Sequential Deep Learning](https://mlsanity.com/paper/2410.06800/) (2024)
- [Continual Learning Beyond Experience Rehearsal and Full Model Surrogates](https://mlsanity.com/paper/2505.21942/) (2025)
- [Global Pre-fixing, Local Adjusting: A Simple yet Effective Contrastive Strategy for Continual Learning](https://mlsanity.com/paper/2509.15347/) (2025)

## Abstract (arXiv)

Catastrophic forgetting is a critical challenge in training deep neural networks. Although continual learning has been investigated as a countermeasure to the problem, it often suffers from the requirements of additional network components and the limited scalability to a large number of tasks. We propose a novel approach to continual learning by approximating a true loss function using an asymmetric quadratic function with one of its sides overestimated. Our algorithm is motivated by the empirical observation that the network parameter updates affect the target loss functions asymmetrically. In the proposed continual learning framework, we estimate an asymmetric loss function for the tasks considered in the past through a proper overestimation of its unobserved sides in training new tasks, while deriving the accurate model parameter for the observable sides. In contrast to existing approaches, our method is free from the side effects and achieves the state-of-the-art accuracy that is even close to the upper-bound performance on several challenging benchmark datasets.