mlsanityData to 7 Oct 2026

Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection

arXiv 2603.19145 ↗First posted 19 Mar 2026cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent paradigms in Random Projection Layer (RPL)-based continual representation learning have demonstrated superior performance when building upon a pre-trained model (PTM). These methods insert a randomly initialized RPL after a PTM to enhance feature representation in the initial stage. Subsequently, a linear classification head is used for analytic updates in the continual learning stage. However, under severe domain gaps between pre-trained representations and target domains, a randomly initialized RPL exhibits limited expressivity under large domain shifts. While largely scaling up the RPL dimension can improve expressivity, it also induces an ill-conditioned feature matrix, thereby destabilizing the recursive analytic updates of the linear head. To this end, we propose the Stochastic Continual Learner with MemoryGuard Supervisory Mechanism (SCL-MGSM). Unlike random initialization, MGSM constructs the projection layer via a principled, data-guided mechanism that progressively selects target-aligned random bases to adapt the PTM representation to downstream tasks. This facilitates the construction of a compact yet expressive RPL while improving the numerical stability of analytic updates. Extensive experiments on multiple exemplar-free Class Incremental Learning (CIL) benchmarks demonstrate that SCL-MGSM achieves superior performance compared to state-of-the-art methods.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

30 of 42 references.

References · 30

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  4. 2022Visual Prompt Tuningcited by 1,674
  5. 2019Natural Adversarial Examplescited by 1,401
  6. 2021Learning to Prompt for Continual Learningcited by 938
  7. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  8. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  9. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  10. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  11. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  12. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
18 more
  1. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  2. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  3. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
  4. 2022ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protectioncited by 56
  5. 2023First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learningcited by 46
  6. 2022Benchmarking Omni-Vision Representation through the Lens of Visual Realmscited by 38
  7. 2024MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learningcited by 35
  8. 2017Stochastic Configuration Networks: Fundamentals and Algorithmscited by 34
  9. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  10. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  11. 2024DiffClass: Diffusion-Based Class Incremental Learningcited by 23
  12. 2025Structural features of the fly olfactory circuit mitigate the stability-plasticity dilemma in continual learningcited by 14
  13. 2024Continual Learning Using a Kernel-Based Method Over Foundation Modelscited by 13
  14. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 10
  15. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  16. 2025AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projectioncited by 7
  17. 2024Harnessing Neural Unit Dynamics for Effective and Scalable Class-Incremental Learningcited by 4
  18. 2025Achieving More with Less: Additive Prompt Tuning for Rehearsal-Free Class-Incremental Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  2. 2026Representation Collapse in Sequential Post-Training of Large Language Modelscited by 2
  3. 2026The Long-Term Effects of Data Selection in LLM Fine-Tuningcited by 1
  4. 2026SPARCL: Spectral Partitioned Analytic Continual Learningcited by 0