# Bypassing Logits Bias in Online Class-Incremental Learning with a Generative Framework

arXiv: https://arxiv.org/abs/2205.09347 · HTML: https://mlsanity.com/paper/2205.09347/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 25 of 25)

- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Semantic Drift Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2004.00440/) (2020)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams](https://mlsanity.com/paper/2009.00919/) (2020)
- [A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning](https://mlsanity.com/paper/2001.00689/) (2020)
- [A continual learning survey: Defying forgetting in classification tasks](https://mlsanity.com/paper/1909.08383/) (2019)
- [A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses](https://mlsanity.com/paper/2003.08983/) (2020)
- [Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates](https://mlsanity.com/paper/2010.00373/) (2020)
- [Generative and Discriminative Text Classification with Recurrent Neural Networks](https://mlsanity.com/paper/1703.01898/) (2017)
- [Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference](https://mlsanity.com/paper/2010.03760/) (2020)

## Abstract (arXiv)

Continual learning requires the model to maintain the learned knowledge while learning from a non-i.i.d data stream continually. Due to the single-pass training setting, online continual learning is very challenging, but it is closer to the real-world scenarios where quick adaptation to new data is appealing. In this paper, we focus on online class-incremental learning setting in which new classes emerge over time. Almost all existing methods are replay-based with a softmax classifier. However, the inherent logits bias problem in the softmax classifier is a main cause of catastrophic forgetting while existing solutions are not applicable for online settings. To bypass this problem, we abandon the softmax classifier and propose a novel generative framework based on the feature space. In our framework, a generative classifier which utilizes replay memory is used for inference, and the training objective is a pair-based metric learning loss which is proven theoretically to optimize the feature space in a generative way. In order to improve the ability to learn new data, we further propose a hybrid of generative and discriminative loss to train the model. Extensive experiments on several benchmarks, including newly introduced task-free datasets, show that our method beats a series of state-of-the-art replay-based methods with discriminative classifiers, and reduces catastrophic forgetting consistently with a remarkable margin.