PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA
Not on the map. Why?
Abstract (arXiv)
Applying differential privacy (DP) via DP-SGD to Low-Rank Adaptation (LoRA) is a natural approach for privacy-preserving fine-tuning. However, LoRA's low-rank parameterization poses a fundamental challenge. In LoRA, each trainable update is represented as a low-rank matrix Z = AB^top, but this factorization is inherently non-identifiable: many factor pairs (A,B) represent the same update Z. As a result, applying DP-SGD directly to the factors induces gauge-dependent perturbations on Z, and we show that this naive DP-LoRA can lead to unbounded noise amplification. We propose PRISM, an intrinsic DP mechanism for LoRA that is gauge invariant by construction, avoids bilinear noise amplification, and admits an efficient low-dimensional noise sampler. Moreover, PRISM yields a closed-form characterization of the effective intrinsic noise induced on Z, enabling stable privacy-utility trade-offs through bounded, gauge-invariant perturbations. We establish standard (ε,δ)-DP guarantees for PRISM and introduce a DP-aware, gauge-invariant adaptive update rule that prevents adaptive optimization from amplifying injected privacy noise, improving numerical stability in practice.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 31 May 2026.
Builds on
30 of 44 references.
References · 30
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2016Deep Learning with Differential Privacycited by 2,728
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2025Gemma 3 Technical Reportcited by 1,824
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
- 2019Machine Unlearningcited by 1,061
18 more
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
- 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
- 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
- 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
- 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
- 2021Differentially Private Fine-tuning of Language Modelscited by 400
- 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
- 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
- 2021Numerical Composition of Differential Privacycited by 180
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
- 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
- 2022Private Adaptive Optimization with Side Informationcited by 38
- 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 32
- 2023DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)cited by 28
- 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
- 2025DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Modelscited by 4
Built on by
Not cited yet.