# Stable-Plastic Subspace Construction for Differentially Private Continual LoRA

arXiv: https://arxiv.org/abs/2610.17904 · HTML: https://mlsanity.com/paper/2610.17904/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 30 of 32)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning](https://mlsanity.com/paper/2404.19245/) (2024)
- [Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration](https://mlsanity.com/paper/2311.06062/) (2023)
- [Asymmetry in Low-Rank Adapters of Foundation Models](https://mlsanity.com/paper/2402.16842/) (2024)
- [SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning](https://mlsanity.com/paper/2501.13198/) (2025)
- [Consistent Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2403.08568/) (2024)
- [Differentially Private Bias-Term Fine-tuning of Foundation Models](https://mlsanity.com/paper/2210.00036/) (2022)
- [CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2505.24816/) (2025)
- [KeepLoRA: Continual Learning with Residual Gradient Adaptation](https://mlsanity.com/paper/2601.19659/) (2026)
- [Differentially Private Fine-Tuning of Diffusion Models](https://mlsanity.com/paper/2406.01355/) (2024)
- [SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models](https://mlsanity.com/paper/2411.02175/) (2024)
- [LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2503.18985/) (2025)
- [Merge before Forget: A Single LoRA Continual Learning via Continual Merging](https://mlsanity.com/paper/2512.23017/) (2025)
- [SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting](https://mlsanity.com/paper/2505.22370/) (2025)
- [Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning](https://mlsanity.com/paper/2603.00191/) (2026)
- [Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning](https://mlsanity.com/paper/2502.07560/) (2025)
- [PLAN: Proactive Low-Rank Allocation for Continual Learning](https://mlsanity.com/paper/2510.21188/) (2025)
- [Window-based Membership Inference Attacks Against Fine-tuned Large Language Models](https://mlsanity.com/paper/2601.02751/) (2026)
- [Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models](https://mlsanity.com/paper/2602.19926/) (2026)
- [Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2407.17120/) (2024)

## Abstract (arXiv)

Continual learning (CL) adapts pretrained models to a sequence of tasks while retaining knowledge acquired from earlier tasks. When task data are sensitive, differential privacy (DP) provides record-level protection, but maintaining both adaptation and retention under privacy constraints remains challenging. We study this problem in continual low-rank adaptation (LoRA) and observe that: 1) private training increases forgetting primarily during subsequent adaptation rather than initial task acquisition; 2) jointly perturbing both LoRA factors introduces a non-negligible bilinear noise interaction; and 3) DP aggravates the stability-plasticity dilemma, hindering both knowledge retention and adaptation to new tasks. Based on these observations, we propose DP-SPA, which constructs a stable subspace from previously privatized updates and a plastic subspace from pretrained weights within its orthogonal complement. The method fixes these bases and privately optimizes only the LoRA factor, eliminating the bilinear noise interaction within each step. Before merging each task update, it rescales stable-direction coefficients using historical and current update energies. We establish a privacy guarantee and show that recalibration does not increase historical overlap and tightens a conditional upper bound on old-task loss increase. Experiments across five datasets, eight settings, and multiple privacy budgets show that DP-SPA improves average accuracy and reduces forgetting compared with the evaluated baselines.