# Continual Learning Using a Kernel-Based Method Over Foundation Models

arXiv: https://arxiv.org/abs/2412.15571 · HTML: https://mlsanity.com/paper/2412.15571/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [RanPAC: Random Projections and Pre-trained Models for Continual Learning](https://mlsanity.com/paper/2307.02251/) (2023)
- [LAMOL: LAnguage MOdeling for Lifelong Language Learning](https://mlsanity.com/paper/1909.03329/) (2019)
- [A Large-Scale Car Dataset for Fine-Grained Categorization and Verification](https://mlsanity.com/paper/1506.08959/) (2015)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Continual Learning for Text Classification with Information Disentanglement Based Regularization](https://mlsanity.com/paper/2104.05489/) (2021)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [ELLE: Efficient Lifelong Pre-training for Emerging Data](https://mlsanity.com/paper/2203.06311/) (2022)
- [Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer](https://mlsanity.com/paper/2211.00789/) (2022)
- [Recent Advances of Foundation Language Models-based Continual Learning: A Survey](https://mlsanity.com/paper/2405.18653/) (2024)
- [Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking](https://mlsanity.com/paper/2107.08173/) (2021)
- [Class Incremental Learning via Likelihood Ratio Based Task Prediction](https://mlsanity.com/paper/2309.15048/) (2023)
- [Lifelong Intent Detection via Multi-Strategy Rebalancing](https://mlsanity.com/paper/2108.04445/) (2021)
- [Class-Incremental Learning based on Label Generation](https://mlsanity.com/paper/2306.12619/) (2023)

## Built on by (most cited first, 14 of 14)

- [AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection](https://mlsanity.com/paper/2511.13880/) (2025)
- [Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages](https://mlsanity.com/paper/2603.12658/) (2026)
- [Achieving Upper Bound Accuracy of Joint Training in Continual Learning](https://mlsanity.com/paper/2502.12388/) (2025)
- [Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection](https://mlsanity.com/paper/2603.19145/) (2026)
- [Continual Knowledge Consolidation LORA for Domain Incremental Learning](https://mlsanity.com/paper/2510.16077/) (2025)
- [HyCal: A Training-Free Prototype Calibration Method for Cross-Discipline Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2604.15678/) (2026)
- [Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL](https://mlsanity.com/paper/2508.08677/) (2025)
- [Source-Free Cross-Domain Continual Learning](https://mlsanity.com/paper/2510.01649/) (2025)
- [Learning After Model Deployment](https://mlsanity.com/paper/2510.17160/) (2025)
- [Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities](https://mlsanity.com/paper/2605.20372/) (2026)
- [Spectral-Aware Analytic Class-Incremental Learning for Long-Tailed Distributions](https://mlsanity.com/paper/2607.22931/) (2026)
- [Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning](https://mlsanity.com/paper/2607.23837/) (2026)
- [Visual Branch is What You Need for CLIP-based Class-Incremental Learning](https://mlsanity.com/paper/2609.37888/) (2026)
- [SPARS: A Stability-Plasticity-Aware Region Selection Framework for Continual Learning](https://mlsanity.com/paper/2610.19876/) (2026)

## Abstract (arXiv)

Continual learning (CL) learns a sequence of tasks incrementally. This paper studies the challenging CL setting of class-incremental learning (CIL). CIL has two key challenges: catastrophic forgetting (CF) and inter-task class separation (ICS). Despite numerous proposed methods, these issues remain persistent obstacles. This paper proposes a novel CIL method, called Kernel Linear Discriminant Analysis (KLDA), that can effectively avoid CF and ICS problems. It leverages only the powerful features learned in a foundation model (FM). However, directly using these features proves suboptimal. To address this, KLDA incorporates the Radial Basis Function (RBF) kernel and its Random Fourier Features (RFF) to enhance the feature representations from the FM, leading to improved performance. When a new task arrives, KLDA computes only the mean for each class in the task and updates a shared covariance matrix for all learned classes based on the kernelized features. Classification is performed using Linear Discriminant Analysis. Our empirical evaluation using text and image classification datasets demonstrates that KLDA significantly outperforms baselines. Remarkably, without relying on replay data, KLDA achieves accuracy comparable to joint training of all classes, which is considered the upper bound for CIL performance. The KLDA code is available at https://github.com/salehmomeni/klda.