mlsanityData to 14 Oct 2026

The Privacy-Hallucination Tradeoff in Differentially Private Language Models

arXiv 2609.00492 ↗First posted 31 Aug 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Both privacy and factual accuracy are paramount in high-stakes domains like healthcare. Concerningly, we uncover and investigate a privacy-hallucination tradeoff in differentially private (DP) language models. First, we empirically show that models pre-trained or fine-tuned with DP tend to produce more hallucinations than non-DP counterparts, with increased severity as the privacy budget grows stricter. Second, we investigate model properties driving this tradeoff, demonstrating that DP mechanisms flatten output distributions, potentially redistributing probability mass toward factually incorrect alternatives. Third, through experiments where we control fact frequency in training data, we characterize how information frequency can reduce hallucination risks in DP models. Overall, our findings underscore the need for more nuanced privacy-preserving interventions that offer rigorous privacy guarantees without compromising factual accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 72 of its 92 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

72 of 92 references.

References · 72

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  7. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  8. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  9. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  10. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  11. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  12. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
38 more
  1. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  2. 2023Fine-tuning Language Models for Factualitycited by 221
  3. 2025Why Language Models Hallucinatecited by 220
  4. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 175
  5. 2024Long-form factuality in large language modelscited by 155
  6. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  7. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  8. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  9. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  10. 2024On Protecting the Data Privacy of Large Language Models (LLMs): A Surveycited by 96
  11. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  12. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 88
  13. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 79
  14. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  15. 2020Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairnesscited by 67
  16. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 66
  17. 2024Factuality of Large Language Models: A Surveycited by 66
  18. 2023Harnessing large-language models to generate private synthetic textcited by 65
  19. 2020Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacycited by 61
  20. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Modelscited by 56
  21. 2022Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributionscited by 52
  22. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 51
  23. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  24. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  25. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 45
  26. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  27. 2024Privacy-Preserving Retrieval-Augmented Generation with Differential Privacycited by 39
  28. 2023User Inference Attacks on Large Language Modelscited by 36
  29. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 36
  30. 2024Scaling Laws for Fact Memorization of Large Language Modelscited by 33
  31. 2021Decision Making with Differential Privacy under a Fairness Lenscited by 32
  32. 2023Diving Deep into Modes of Fact Hallucinations in Dialogue Systemscited by 32
  33. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  34. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 30
  35. 2022DP-Rewrite: Towards Reproducibility and Transparency in Differentially Private Text Rewritingcited by 28
  36. 2023Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deploymentcited by 27
  37. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  38. 2025Scaling Laws for Differentially Private Language Modelscited by 22

50 most cited of 72 All 72 (JSON)

Built on by

Not cited yet.