# Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learning

arXiv: https://arxiv.org/abs/2507.04020 · HTML: https://mlsanity.com/paper/2507.04020/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 5 Jul 2025.

## Builds on (most cited first, 30 of 34)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2004.10956/) (2020)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Towards Robust Vision Transformer](https://mlsanity.com/paper/2105.07926/) (2021)
- [Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges](https://mlsanity.com/paper/1907.00182/) (2019)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [Reveal of Vision Transformers Robustness against Adversarial Attacks](https://mlsanity.com/paper/2106.03734/) (2021)
- [Understanding Continual Learning Settings with Data Distribution Drift Analysis](https://mlsanity.com/paper/2104.01678/) (2021)
- [CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks](https://mlsanity.com/paper/2112.02714/) (2021)
- [Danish Fungi 2020 -- Not Just Another Image Recognition Dataset](https://mlsanity.com/paper/2103.10107/) (2021)
- [AFR-Net: Attention-Driven Fingerprint Recognition Network](https://mlsanity.com/paper/2211.13897/) (2022)
- [Is Class-Incremental Enough for Continual Learning?](https://mlsanity.com/paper/2112.02925/) (2021)
- [Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey](https://mlsanity.com/paper/2408.07583/) (2024)
- [Continual Learning of Diffusion Models with Generative Distillation](https://mlsanity.com/paper/2311.14028/) (2023)
- [On the Kolmogorov neural networks](https://mlsanity.com/paper/2311.00049/) (2023)
- [A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships](https://mlsanity.com/paper/2408.15178/) (2024)
- [MoViT: Memorizing Vision Transformers for Medical Image Analysis](https://mlsanity.com/paper/2303.15553/) (2023)
- [Multi-View Class Incremental Learning](https://mlsanity.com/paper/2306.09675/) (2023)
- [Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training](https://mlsanity.com/paper/2412.12496/) (2024)
- [Surface Analysis with Vision Transformers](https://mlsanity.com/paper/2205.15836/) (2022)
- [Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations](https://mlsanity.com/paper/2211.08794/) (2022)
- [Lossless Token Merging Even Without Fine-Tuning in Vision Transformers](https://mlsanity.com/paper/2505.15160/) (2025)
- [Adversarial Learning of General Transformations for Data Augmentation](https://mlsanity.com/paper/1909.09801/) (2019)

## Built on by (most cited first, 1 of 1)

- [Hybrid Parallel Bessel KAN Model: a Kolmogorov Arnold Network: integrated framework for developing a novel architecture to reconstruct the fused-basis approach via a parameter free quadratic interaction](https://mlsanity.com/paper/2610.17779/) (2026)

## Abstract (arXiv)

Continual learning (CL), the ability of a model to learn new tasks without forgetting previously acquired knowledge, remains a critical challenge in artificial intelligence, particularly for vision transformers (ViTs) utilizing Multilayer Perceptrons (MLPs) for global representation learning. Catastrophic forgetting, where new information overwrites prior knowledge, is especially problematic in these models. This research proposes replacing MLPs in ViTs with Kolmogorov-Arnold Network (KANs) to address this issue. KANs leverage local plasticity through spline-based activations, ensuring that only a subset of parameters is updated per sample, thereby preserving previously learned knowledge. The study investigates the efficacy of KAN-based ViTs in CL scenarios across benchmark datasets (MNIST, CIFAR100), focusing on their ability to retain accuracy on earlier tasks while adapting to new ones. Experimental results demonstrate that KAN-based ViTs significantly mitigate catastrophic forgetting, outperforming traditional MLP-based ViTs in knowledge retention and task adaptation. This novel integration of KANs into ViTs represents a promising step toward more robust and adaptable models for dynamic environments.