# Neural Architecture for Online Ensemble Continual Learning

arXiv: https://arxiv.org/abs/2211.14963 · HTML: https://mlsanity.com/paper/2211.14963/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 27 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning](https://mlsanity.com/paper/2001.00689/) (2020)
- [Neural Nearest Neighbors Networks](https://mlsanity.com/paper/1810.12575/) (2018)
- [A Survey on Ensemble Learning under the Era of Deep Learning](https://mlsanity.com/paper/2101.08387/) (2021)
- [CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions](https://mlsanity.com/paper/2009.09929/) (2020)
- [Encoders and Ensembles for Task-Free Continual Learning](https://mlsanity.com/paper/2105.13327/) (2021)
- [A Novel Ensemble Deep Learning Model for Stock Prediction Based on Stock Prices and News](https://mlsanity.com/paper/2007.12620/) (2020)
- [Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees](https://mlsanity.com/paper/1702.08833/) (2017)

## Built on by (most cited first, 1 of 1)

- [Continual learning with the neural tangent ensemble](https://mlsanity.com/paper/2408.17394/) (2024)

## Abstract (arXiv)

Continual learning with an increasing number of classes is a challenging task. The difficulty rises when each example is presented exactly once, which requires the model to learn online. Recent methods with classic parameter optimization procedures have been shown to struggle in such setups or have limitations like non-differentiable components or memory buffers. For this reason, we present the fully differentiable ensemble method that allows us to efficiently train an ensemble of neural networks in the end-to-end regime. The proposed technique achieves SOTA results without a memory buffer and clearly outperforms the reference methods. The conducted experiments have also shown a significant increase in the performance for small ensembles, which demonstrates the capability of obtaining relatively high classification accuracy with a reduced number of classifiers.