mlsanityData to 14 Oct 2026

Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models

arXiv 2509.06100 ↗First posted 7 Sep 2025cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) suffer from catastrophic forgetting in sequential multi-task learning. Existing parameter regularization methods (e.g., O-LoRA, N-LoRA) mitigate interference via low-rank subspace orthogonality, but additive updates distort the intrinsic geometry of model parameters. We propose OLieRA, a Lie group based fine-tuning framework that preserves parameter geometry through multiplicative updates while enforcing orthogonality across task subspaces. OLieRA achieves state-of-the-art performance on the Standard CL benchmark and remains highly competitive under large task sequences. It further inherits the replay-free and task-ID free inference properties of O-LoRA, establishing a principled paradigm for continual learning in LLMs.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 7 Sep 2025.

Builds on

20 of 33 references.

References · 20

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2016Learning without Forgettingcited by 2,496
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  8. 2021Learning to Prompt for Continual Learningcited by 944
  9. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  10. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  11. 2019Episodic Memory in Lifelong Language Learningcited by 188
  12. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
8 more
  1. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 104
  2. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  3. 2024Scalable Language Model with Generalized Continual Learningcited by 28
  4. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  5. 2024Is Parameter Collision Hindering Continual Learning in LLMs?cited by 15
  6. 2021Optimization and Sampling Under Continuous Symmetry: Examples and Lie Theorycited by 5
  7. 2025Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformationscited by 5
  8. 2020Computing Representations for Lie Algebraic Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learningcited by 4
  2. 2026Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptationcited by 4
  3. 2026Memory for Large Language Modelscited by 4
  4. 2026Continual Fine-Tuning of Large Language Models via Program Memorycited by 1
  5. 2025Deep Generative Continual Learning using Functional LoRA: FunLoRAcited by 0
  6. 2026GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretrainingcited by 0
  7. 2026Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectificationcited by 0