mlsanityData to 14 Oct 2026

Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA

arXiv 2506.20856 ↗First posted 25 Jun 2025cs.LG cs.CL cs.CRcited by 3

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Memorization in large language models (LLMs) makes them vulnerable to data extraction attacks. While pre-training memorization has been extensively studied, fewer works have explored its impact in fine-tuning, particularly for LoRA fine-tuning, a widely adopted parameter-efficient method. In this work, we re-examine memorization in fine-tuning and uncover a surprising divergence from prior findings across different fine-tuning strategies. Factors such as model scale and data duplication, which strongly influence memorization in pre-training and full fine-tuning, do not follow the same trend in LoRA fine-tuning. Using a more relaxed similarity-based memorization metric, we demonstrate that LoRA significantly reduces memorization risks compared to full fine-tuning, while still maintaining strong task performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 9 of its 24 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

9 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Rethinking LLM Memorization through the Lens of Adversarial Compression23 Apr

Other references · 8

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  3. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  5. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 175
  6. 2023Exploring Memorization in Fine-tuned Language Modelscited by 45
  7. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36
  8. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Decomposing Memorization Reduction in Privacy-Preserving Fine-Tuning of SLMs for CSIRTscited by 1
  2. 2026Not All 4-bit Quantizers Are Equal: Deployment-Time Mitigation of PII Leakage in Fine-Tuned Small Language Modelscited by 0
  3. 2026SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Memorization in Fine-Tuned Large Language Models
  2. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Models
  3. 2025Impact of Fine-Tuning Methods on Memorization in Large Language Models
  4. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation
  5. 2022Memorization in NLP Fine-tuning Methods
  6. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks
  7. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  8. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?