mlsanityData to 14 Oct 2026

OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning

arXiv 2510.13003 ↗First posted 14 Oct 2025cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models but suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions that encode essential pre-trained knowledge. We propose Orthogonal Projection LoRA (OPLoRA), a theoretically grounded approach that prevents this interference through double-sided orthogonal projections. By decomposing frozen weights via SVD, OPLoRA constrains LoRA updates to lie entirely within the orthogonal complement of the top-k singular subspace using projections PL = I - Uk Uk^top and PR = I - Vk Vk^top. We prove that this construction exactly preserves the top-k singular triples, providing mathematical guarantees for knowledge retention. To quantify subspace interference, we introduce ρk, a metric measuring update alignment with dominant directions. Extensive experiments across commonsense reasoning, mathematics, and code generation demonstrate that OPLoRA significantly reduces forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B, establishing orthogonal projection as an effective mechanism for knowledge preservation in parameter-efficient fine-tuning.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

22 of 27 references.

References · 22

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2016Learning without Forgettingcited by 2,496
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
10 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  3. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  5. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  6. 2024LoRA Learns Less and Forgets Lesscited by 328
  7. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  8. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  9. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  10. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 18.

21 papers.

Sort:

By area:

  1. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  2. 2026Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learningcited by 5
  3. 2025The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Modelscited by 2
  4. 2026The Intruder Threshold: A Spectral Law for LoRA Fine-Tuningcited by 2
  5. 2026OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWebcited by 1
  6. 2026Unsupervised Adaptation of PDE Foundation Modelscited by 1
  7. 2026Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Modelscited by 1
  8. 2025NP-LoRA: Null Space Projection for Subject-Style LoRA Fusioncited by 0
  9. 2025R2R: A Route-to-Rerank Post-Training Framework for Multi-Domain Decoder-Only Rerankerscited by 0
  10. 2026Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mappingcited by 0
  11. 2026GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretrainingcited by 0
  12. 2026PALoRA: Projection-Adaptive LoRA for Preserving Reasoning in Large Language Modelscited by 0
9 more
  1. 2026Foundation-Preserving Optimization in Generalized Eigenspacecited by 0
  2. 2026CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMscited by 0
  3. 2026Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearningcited by 0
  4. 2026Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuningcited by 0
  5. 2026FADRA: Frequency-Aware Diffusion with Residual Adaptation for Video Face Restorationcited by 0
  6. 2026CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Frameworkcited by 0
  7. 2026Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlasescited by 0
  8. 2026Local Support Learningcited by 0
  9. 2026Geometric Calibration of Supervised Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TLoRA: Task-aware Low Rank Adaptation of Large Language Models
  2. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  3. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
  4. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
  5. 2025Uni-LoRA: One Vector is All You Need
  6. 2025DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  7. 2026Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models
  8. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Models