# Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD

arXiv: https://arxiv.org/abs/2601.10045 · HTML: https://mlsanity.com/paper/2601.10045/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 15 Jan 2026.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 21 of 21)

- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Low-Cost High-Power Membership Inference Attacks](https://mlsanity.com/paper/2312.03262/) (2023)
- [Fingerprinting Codes and the Price of Approximate Differential Privacy](https://mlsanity.com/paper/1311.3158/) (2013)
- [Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy](https://mlsanity.com/paper/2205.10683/) (2022)
- [Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clipping](https://mlsanity.com/paper/2009.03106/) (2020)
- [Implicit Regularization in Tensor Factorization](https://mlsanity.com/paper/2102.09972/) (2021)
- [PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Traps](https://mlsanity.com/paper/2403.09562/) (2024)
- [Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs](https://mlsanity.com/paper/2408.01008/) (2024)
- [LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models](https://mlsanity.com/paper/2507.18302/) (2025)
- [TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts](https://mlsanity.com/paper/2504.21190/) (2025)
- [LoRA Provides Differential Privacy by Design via Random Sketching](https://mlsanity.com/paper/2409.17538/) (2024)

## Abstract (arXiv)

Fine-tuning large language models on sensitive data poses significant privacy risks, as membership inference attacks can reveal whether individual records were used during training. While Differential Privacy (DP) provides formal protection, applying DP to conventional Parameter-Efficient Fine-Tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) often incurs substantial utility loss. In this work, we show that a more structurally constrained PEFT architecture, Tensor Train Low-Rank Adaptation (TTLoRA), can improve the privacy-utility tradeoff by shrinking the effective parameter space while preserving expressivity. To this end, we develop TTLoRA-DP, a differentially private training framework for TTLoRA. Specifically, we extend the ghost clipping algorithm to Tensor Train cores via cached contraction states, enabling efficient Differentially Private Stochastic Gradient Descent (DP-SGD) with exact per-example gradient norm computation without materializing full per-example gradients. Experiments on GPT-2 fine-tuning over the Enron and Penn Treebank datasets show that TTLoRA-DP consistently strengthens privacy protection relative to LoRA-DP while maintaining comparable or better downstream utility. Moreover, TTLoRA exhibits lower membership leakage even without DP training, using substantially smaller adapters and requiring on average 7.6X fewer parameters than LoRA. Overall, our results demonstrate that TTLoRA offers a practical path to improving the privacy-utility tradeoff in parameter-efficient language model adaptation.