mlsanityData to 14 Oct 2026

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

arXiv 2604.12526 ↗First posted 14 Apr 2026cs.LG cs.AIcited by 0

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else. This setting becomes especially difficult when deletion requests arrive sequentially, because the model must repeatedly adapt without erasing previously retained knowledge. Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision, causing textitstrong interference between tasks. We propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection. Our method constrains each new LoRA update during training so that it lies in the orthogonal complement of the subspaces used by earlier unlearning tasks. This preserves task isolation without requiring dynamic routing at deployment. Experiments on CIFAR-100 with ResNet-20 and on MNIST show stable behavior across long sequences of unlearning tasks. After thirty sequential unlearning tasks, state-of-the-art static fusion reduces retained accuracy from 60.39% to 12.70%, whereas the proposed in-training constrained optimization maintains baseline performance (~58.1%) while preserving strong unlearning efficacy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 8 of its 11 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

8 of 11 references.

References · 8

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2019Machine Unlearningcited by 1,061
  6. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  7. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 179
  8. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
  2. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  3. 2025Uni-LoRA: One Vector is All You Need
  4. 2026FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
  5. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning
  6. 2024Machine Unlearning on Pre-trained Models by Residual Feature Alignment Using LoRA
  7. 2026Retain-Neutral Surrogates for Min-Max Unlearning
  8. 2026BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning