# Task agnostic continual learning with Pairwise layer architecture

arXiv: https://arxiv.org/abs/2405.13632 · HTML: https://mlsanity.com/paper/2405.13632/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 30 of 31)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines](https://mlsanity.com/paper/1810.12488/) (2018)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [A Study on Overfitting in Deep Reinforcement Learning](https://mlsanity.com/paper/1804.06893/) (2018)
- [Uncertainty-based Continual Learning with Adaptive Regularization](https://mlsanity.com/paper/1905.11614/) (2019)
- [Uncertainty-guided Continual Learning with Bayesian Neural Networks](https://mlsanity.com/paper/1906.02425/) (2019)
- [Rehearsal revealed: The limits and merits of revisiting samples in continual learning](https://mlsanity.com/paper/2104.07446/) (2021)
- [Task Agnostic Continual Learning Using Online Variational Bayes](https://mlsanity.com/paper/1803.10123/) (2018)
- [Selfless Sequential Learning](https://mlsanity.com/paper/1806.05421/) (2018)
- [AutoCross: Automatic Feature Crossing for Tabular Data in Real-World Applications](https://mlsanity.com/paper/1904.12857/) (2019)
- [Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments](https://mlsanity.com/paper/2201.00042/) (2021)
- [Task-Free Continual Learning via Online Discrepancy Distance Learning](https://mlsanity.com/paper/2210.06579/) (2022)
- [Continual Learning in Deep Networks: an Analysis of the Last Layer](https://mlsanity.com/paper/2106.01834/) (2021)
- [Prediction Error-based Classification for Class-Incremental Learning](https://mlsanity.com/paper/2305.18806/) (2023)
- [Algorithmic insights on continual learning from fruit flies](https://mlsanity.com/paper/2107.07617/) (2021)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)

## Abstract (arXiv)

Most of the dominant approaches to continual learning are based on either memory replay, parameter isolation, or regularization techniques that require task boundaries to calculate task statistics. We propose a static architecture-based method that doesn't use any of these. We show that we can improve the continual learning performance by replacing the final layer of our networks with our pairwise interaction layer. The pairwise interaction layer uses sparse representations from a Winner-take-all style activation function to find the relevant correlations in the hidden layer representations. The networks using this architecture show competitive performance in MNIST and FashionMNIST-based continual image classification experiments. We demonstrate this in an online streaming continual learning setup where the learning system cannot access task labels or boundaries.