mlsanityData to 14 Oct 2026

Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation

arXiv 2411.05663 ↗First posted 8 Nov 2024cs.CV cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Catastrophic forgetting is a significant challenge in online continual learning (OCL), especially for non-stationary data streams that do not have well-defined task boundaries. This challenge is exacerbated by the memory constraints and privacy concerns inherent in rehearsal buffers. To tackle catastrophic forgetting, in this paper, we introduce Online-LoRA, a novel framework for task-free OCL. Online-LoRA allows to finetune pre-trained Vision Transformer (ViT) models in real-time to address the limitations of rehearsal buffers and leverage pre-trained models' performance benefits. As the main contribution, our approach features a novel online weight regularization strategy to identify and consolidate important model parameters. Moreover, Online-LoRA leverages the training dynamics of loss values to enable the automatic recognition of the data distribution shifts. Extensive experiments across many task-free OCL scenarios and benchmark datasets (including CIFAR-100, ImageNet-R, ImageNet-S, CUB-200 and CORe50) demonstrate that Online-LoRA can be robustly adapted to various ViT architectures, while achieving better performance compared to SOTA methods. Our code will be publicly available at: https://github.com/Christina200/Online-LoRA-official.git.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 78 of its 103 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

78 of 103 references.

References · 78

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  12. 2022Visual Prompt Tuningcited by 1,681
38 more
  1. 2016Progressive Neural Networkscited by 1,361
  2. 2017Continual Learning with Deep Generative Replaycited by 1,158
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  4. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  6. 2021Learning to Prompt for Continual Learningcited by 944
  7. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  8. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  9. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  10. 2019Large Scale Incremental Learningcited by 732
  11. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  12. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  13. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  14. 2018End-to-End Incremental Learningcited by 597
  15. 2019Gradient based sample selection for online continual learningcited by 571
  16. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  17. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
  18. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  19. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  20. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  21. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  22. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 289
  23. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  24. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  25. 2021Co2L: Contrastive Continual Learningcited by 244
  26. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 229
  27. 2018Task-Free Continual Learningcited by 221
  28. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  29. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  30. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  31. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 180
  32. 2018Towards Robust Evaluations of Continual Learningcited by 148
  33. 2019Continual Unsupervised Representation Learningcited by 147
  34. 2020Continual Prototype Evolution: Learning Online from Non-Stationary Data Streamscited by 144
  35. 2020Adversarial Continual Learningcited by 142
  36. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 141
  37. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  38. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 134

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 19.

27 papers.

Sort:

By area:

  1. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  2. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 28
  3. 2025PLAN: Proactive Low-Rank Allocation for Continual Learningcited by 7
  4. 2026Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systemscited by 7
  5. 2025C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Modelscited by 3
  6. 2026Revisiting Weight Regularization for Low-Rank Continual Learningcited by 3
  7. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Modelscited by 1
  8. 2026Live Interactive Training for Video Segmentationcited by 1
  9. 2026Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learningcited by 1
  10. 2026Is Prompt Selection Necessary for Task-Free Online Continual Learning?cited by 1
  11. 2026ImageHD: Energy-Efficient On-Device Continual Learning of Visual Representations via Hyperdimensional Computingcited by 1
  12. 2026Normality-Preserving Continual Industrial Anomaly Detection via Orthogonal LoRA Bankscited by 1
15 more
  1. 2025From Offline to Online Memory-Free and Task-Free Continual Learning via Fine-Grained Hypergradientscited by 0
  2. 2025Are Current Continual Learning Methods Truly Agnostic? Introducing OPRE, a Step Toward Agnostic Continual Learningcited by 0
  3. 2025Dynamic Feedback Engines: Layer-Wise Control for Self-Regulating Continual Learningcited by 0
  4. 2026Routing without Forgettingcited by 0
  5. 2026Continual Fine-Tuning with Provably Accurate and Parameter-Free Task Retrievalcited by 0
  6. 2026Online Continual Learning with Dynamic Label Hierarchiescited by 0
  7. 2026LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Modelscited by 0
  8. 2026APEX: Adaptive Principle EXtraction A Three-Layer Self-Evolution Framework for Production AI Agentscited by 0
  9. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMscited by 0
  10. 2026Learning to Fine-tune Foundation Models under Resource Limitationscited by 0
  11. 2026Parameter Efficient Continual Learning for Sparse Event-Based Transformerscited by 0
  12. 2026Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoringcited by 0
  13. 2026Neural-Collapse-guided Task-Free Continual Anomaly Detectioncited by 0
  14. 2026A Dynamical Theory of LoRA in Continual Learningcited by 0
  15. 2026Formalizing Forgetting: Theory of Forgetting for Online Continual Learningcited by 0