# Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning

arXiv: https://arxiv.org/abs/2509.16738 · HTML: https://mlsanity.com/paper/2509.16738/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 57 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Builds on (most cited first, 30 of 45)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [RanPAC: Random Projections and Pre-trained Models for Continual Learning](https://mlsanity.com/paper/2307.02251/) (2023)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2309.14062/) (2023)
- [RMM: Reinforced Memory Management for Class-Incremental Learning](https://mlsanity.com/paper/2301.05792/) (2023)
- [ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection](https://mlsanity.com/paper/2205.14922/) (2022)
- [Class-Incremental Exemplar Compression for Class-Incremental Learning](https://mlsanity.com/paper/2303.14042/) (2023)

## Built on by (most cited first, 14 of 14)

- [Object-AVEdit: An Object-level Audio-Visual Editing Model](https://mlsanity.com/paper/2510.00050/) (2025)
- [Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts](https://mlsanity.com/paper/2602.03473/) (2026)
- [Enhancing Neural Video Compression of Static Scenes with Positive-Incentive Noise](https://mlsanity.com/paper/2603.06095/) (2026)
- [Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2510.19802/) (2025)
- [GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning](https://mlsanity.com/paper/2511.15256/) (2025)
- [Real-Time Crowd Counting for Embedded Systems with Lightweight Architecture](https://mlsanity.com/paper/2510.13250/) (2025)
- [Multi-Intent Spoken Language Understanding: Methods, Trends, and Challenges](https://mlsanity.com/paper/2512.11258/) (2025)
- [Noise-Robust Tiny Object Localization with Flows](https://mlsanity.com/paper/2601.00617/) (2026)
- [SR2-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning](https://mlsanity.com/paper/2605.07420/) (2026)
- [Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models](https://mlsanity.com/paper/2606.30168/) (2026)
- [Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2607.17593/) (2026)
- [Representation Learning in Diffusion and Flow-based Model: An Application Aspect](https://mlsanity.com/paper/2608.24068/) (2026)
- [One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning](https://mlsanity.com/paper/2608.31096/) (2026)
- [Incentive Noise and Structural Prior Infusion for Multi-modal Object Re-Identification](https://mlsanity.com/paper/2609.24539/) (2026)

## Abstract (arXiv)

Class Incremental Learning (CIL) aims to continuously learn new categories while retaining the knowledge of old ones. Pre-trained models (PTMs) show promising capabilities in CIL. However, existing approaches that apply lightweight fine-tuning to backbones still induce parameter drift, thereby compromising the generalization capability of pre-trained models. Parameter drift can be conceptualized as a form of noise that obscures critical patterns learned for previous tasks. However, recent researches have shown that noise is not always harmful. For example, the large number of visual patterns learned from pre-training can be easily abused by a single task, and introducing appropriate noise can suppress some low-correlation features, thus leaving a margin for future tasks. To this end, we propose learning beneficial noise for CIL guided by information theory and propose Mixture of Noise (Min), aiming to mitigate the degradation of backbone generalization from adapting new tasks. Specifically, task-specific noise is learned from high-dimension features of new tasks. Then, a set of weights is adjusted dynamically for optimal mixture of different task noise. Finally, Min embeds the beneficial noise into the intermediate features to mask the response of inefficient patterns. Extensive experiments on six benchmark datasets demonstrate that Min achieves state-of-the-art performance in most incremental settings, with particularly outstanding results in 50-steps incremental settings. This shows the significant potential for beneficial noise in continual learning. Code is available at https://github.com/ASCIIJK/MiN-NeurIPS2025.