# Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation

arXiv: https://arxiv.org/abs/2411.05663 · HTML: https://mlsanity.com/paper/2411.05663/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 78 of its 103 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 30 of 78)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)

## Built on by (most cited first, 27 of 27)

- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2505.24816/) (2025)
- [PLAN: Proactive Low-Rank Allocation for Continual Learning](https://mlsanity.com/paper/2510.21188/) (2025)
- [Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systems](https://mlsanity.com/paper/2604.20745/) (2026)
- [C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models](https://mlsanity.com/paper/2502.17920/) (2025)
- [Revisiting Weight Regularization for Low-Rank Continual Learning](https://mlsanity.com/paper/2602.17559/) (2026)
- [Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models](https://mlsanity.com/paper/2510.21175/) (2025)
- [Live Interactive Training for Video Segmentation](https://mlsanity.com/paper/2603.26929/) (2026)
- [Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning](https://mlsanity.com/paper/2604.01152/) (2026)
- [Is Prompt Selection Necessary for Task-Free Online Continual Learning?](https://mlsanity.com/paper/2604.04420/) (2026)
- [ImageHD: Energy-Efficient On-Device Continual Learning of Visual Representations via Hyperdimensional Computing](https://mlsanity.com/paper/2604.21280/) (2026)
- [Normality-Preserving Continual Industrial Anomaly Detection via Orthogonal LoRA Banks](https://mlsanity.com/paper/2606.02042/) (2026)
- [From Offline to Online Memory-Free and Task-Free Continual Learning via Fine-Grained Hypergradients](https://mlsanity.com/paper/2502.18762/) (2025)
- [Are Current Continual Learning Methods Truly Agnostic? Introducing OPRE, a Step Toward Agnostic Continual Learning](https://mlsanity.com/paper/2511.08226/) (2025)
- [Dynamic Feedback Engines: Layer-Wise Control for Self-Regulating Continual Learning](https://mlsanity.com/paper/2512.21743/) (2025)
- [Routing without Forgetting](https://mlsanity.com/paper/2603.09576/) (2026)
- [Continual Fine-Tuning with Provably Accurate and Parameter-Free Task Retrieval](https://mlsanity.com/paper/2603.13235/) (2026)
- [Online Continual Learning with Dynamic Label Hierarchies](https://mlsanity.com/paper/2605.11742/) (2026)
- [LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models](https://mlsanity.com/paper/2606.09430/) (2026)
- [APEX: Adaptive Principle EXtraction A Three-Layer Self-Evolution Framework for Production AI Agents](https://mlsanity.com/paper/2606.15363/) (2026)
- [Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs](https://mlsanity.com/paper/2606.20961/) (2026)
- [Learning to Fine-tune Foundation Models under Resource Limitations](https://mlsanity.com/paper/2607.10694/) (2026)
- [Parameter Efficient Continual Learning for Sparse Event-Based Transformers](https://mlsanity.com/paper/2608.26720/) (2026)
- [Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoring](https://mlsanity.com/paper/2608.28237/) (2026)
- [Neural-Collapse-guided Task-Free Continual Anomaly Detection](https://mlsanity.com/paper/2609.03406/) (2026)
- [A Dynamical Theory of LoRA in Continual Learning](https://mlsanity.com/paper/2609.39367/) (2026)
- [Formalizing Forgetting: Theory of Forgetting for Online Continual Learning](https://mlsanity.com/paper/2610.14787/) (2026)

## Abstract (arXiv)

Catastrophic forgetting is a significant challenge in online continual learning (OCL), especially for non-stationary data streams that do not have well-defined task boundaries. This challenge is exacerbated by the memory constraints and privacy concerns inherent in rehearsal buffers. To tackle catastrophic forgetting, in this paper, we introduce Online-LoRA, a novel framework for task-free OCL. Online-LoRA allows to finetune pre-trained Vision Transformer (ViT) models in real-time to address the limitations of rehearsal buffers and leverage pre-trained models' performance benefits. As the main contribution, our approach features a novel online weight regularization strategy to identify and consolidate important model parameters. Moreover, Online-LoRA leverages the training dynamics of loss values to enable the automatic recognition of the data distribution shifts. Extensive experiments across many task-free OCL scenarios and benchmark datasets (including CIFAR-100, ImageNet-R, ImageNet-S, CUB-200 and CORe50) demonstrate that Online-LoRA can be robustly adapted to various ViT architectures, while achieving better performance compared to SOTA methods. Our code will be publicly available at: https://github.com/Christina200/Online-LoRA-official.git.