mlsanityData to 14 Oct 2026

Exploring Memorization in Fine-tuned Language Models

arXiv 2310.06714 ↗First posted 10 Oct 2023cs.AI cs.CL cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown great capabilities in various tasks but also exhibited memorization of training data, raising tremendous privacy and copyright concerns. While prior works have studied memorization during pre-training, the exploration of memorization during fine-tuning is rather limited. Compared to pre-training, fine-tuning typically involves more sensitive data and diverse objectives, thus may bring distinct privacy risks and unique memorization behaviors. In this work, we conduct the first comprehensive analysis to explore language models' (LMs) memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

26 of 33 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
14 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  8. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  9. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  10. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  11. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  12. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  13. 2022Do Language Models Plagiarize?cited by 73
  14. 2022Memorization in NLP Fine-tuning Methodscited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 19; 2026: 11.

45 papers.

Sort:

By area:

  1. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  2. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
  3. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 38
  4. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
  5. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  6. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  7. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
  8. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  9. 2024Measuring memorization in RLHF for code completioncited by 13
  10. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 10
  11. 2025Privacy-Aware RAG: Secure and Isolated Knowledge Retrievalcited by 10
  12. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 10
33 more
  1. 2024The Fault in our Stars: Quality Assessment of Code Generation Benchmarkscited by 9
  2. 2024Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Taskscited by 9
  3. 2025Mitigating Memorization in LLMs using Activation Steeringcited by 8
  4. 2024Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Modelscited by 7
  5. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 7
  6. 2026Window-based Membership Inference Attacks Against Fine-tuned Large Language Modelscited by 7
  7. 2025Large Language Models Penetration in Scholarly Writing and Peer Reviewcited by 6
  8. 2025Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Modelscited by 6
  9. 2024Retrieval Augmented Thought Process for Private Data Handling in Healthcarecited by 5
  10. 2025Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implicationscited by 5
  11. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  12. 2025Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformerscited by 4
  13. 2025How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuningcited by 3
  14. 2025Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRAcited by 3
  15. 2025Tab-MIA: A Benchmark Dataset for Membership Inference Attacks on Tabular Data in LLMscited by 3
  16. 2024Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language modelscited by 2
  17. 2025Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledgecited by 2
  18. 2025Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearningcited by 2
  19. 2026Memorization Dynamics in Knowledge Distillation for Language Modelscited by 2
  20. 2026Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generationcited by 2
  21. 2025LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Datacited by 1
  22. 2025Exploring Cross-Client Memorization of Training Data in Large Language Models for Federated Learningcited by 1
  23. 2025Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalizationcited by 1
  24. 2026Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMscited by 1
  25. 2026GraphSteal: Structural Knowledge Stealing from Graph RAG via Traversal Reconstructioncited by 1
  26. 2024Memorization Capacity for Additive Fine-Tuning with Small ReLU Networkscited by 0
  27. 2025Memorization in Fine-Tuned Large Language Modelscited by 0
  28. 2025EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMscited by 0
  29. 2026Extracting Training Dialogue Data from Large Language Model based Task Botscited by 0
  30. 2026Panoptic Pairwise Distortion Graphcited by 0
  31. 2026Reconstruction of Personally Identifiable Information from Proprietary Data in Supervised Fine-Tuned Modelscited by 0
  32. 2026Extracting Training Data from Diffusion Language Models via Infillingcited by 0
  33. 2026Not All 4-bit Quantizers Are Equal: Deployment-Time Mitigation of PII Leakage in Fine-Tuned Small Language Modelscited by 0