# CATFormer: When Continual Learning Meets Spiking Transformers With Dynamic Thresholds

arXiv: https://arxiv.org/abs/2603.15184 · HTML: https://mlsanity.com/paper/2603.15184/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2203.06359/) (2022)
- [Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization](https://mlsanity.com/paper/1802.01569/) (2018)
- [Learning to Continually Learn](https://mlsanity.com/paper/2002.09571/) (2020)
- [Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning](https://mlsanity.com/paper/2403.18886/) (2024)
- [Efficient Video and Audio processing with Loihi 2](https://mlsanity.com/paper/2310.03251/) (2023)
- [CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2505.24816/) (2025)
- [Biologically Inspired Dynamic Thresholds for Spiking Neural Networks](https://mlsanity.com/paper/2206.04426/) (2022)
- [Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation](https://mlsanity.com/paper/2404.00417/) (2024)
- [Dynamic Integration of Task-Specific Adapters for Class Incremental Learning](https://mlsanity.com/paper/2409.14983/) (2024)
- [Relational Experience Replay: Continual Learning by Adaptively Tuning Task-wise Relationship](https://mlsanity.com/paper/2112.15402/) (2021)
- [Enhancing Efficient Continual Learning with Dynamic Structure Development of Spiking Neural Networks](https://mlsanity.com/paper/2308.04749/) (2023)
- [Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators](https://mlsanity.com/paper/2206.01338/) (2022)
- [Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization](https://mlsanity.com/paper/2412.18177/) (2024)
- [ALADE-SNN: Adaptive Logit Alignment in Dynamically Expandable Spiking Neural Networks for Class Incremental Learning](https://mlsanity.com/paper/2412.12696/) (2024)

## Built on by (most cited first, 1 of 1)

- [Parameter Efficient Continual Learning for Sparse Event-Based Transformers](https://mlsanity.com/paper/2608.26720/) (2026)

## Abstract (arXiv)

Although deep neural networks perform extremely well in controlled environments, they fail in real-world scenarios where data isn't available all at once, and the model must adapt to a new data distribution that may or may not follow the initial distribution. Previously acquired knowledge is lost during subsequent updates based on new data. a phenomenon commonly known as catastrophic forgetting. In contrast, the brain can learn without such catastrophic forgetting, irrespective of the number of tasks it encounters. Existing spiking neural networks (SNNs) for class-incremental learning (CIL) suffer a sharp performance drop as tasks accumulate. We here introduce CATFormer (Context Adaptive Threshold Transformer), a scalable framework that overcomes this limitation. We observe that the key to preventing forgetting in SNNs lies not only in synaptic plasticity but also in modulating neuronal excitability. At the core of CATFormer is the Dynamic Threshold Leaky Integrate-and-Fire (DTLIF) neuron model, which leverages context-adaptive thresholds as the primary mechanism for knowledge retention. This is paired with a Gated Dynamic Head Selection (G-DHS) mechanism for task-agnostic inference. Extensive evaluation on both static (CIFAR-10/100/Tiny-ImageNet) and neuromorphic (CIFAR10-DVS/SHD) datasets reveals that CATFormer outperforms existing rehearsal-free CIL algorithms across various task splits, establishing it as an ideal architecture for energy-efficient, true-class incremental learning.