mlsanityData to 7 Oct 2026

Analysis of Privacy Leakage in Federated Large Language Models

arXiv 2403.04784 ↗First posted 2 Mar 2024cs.CR cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

With the rapid adoption of Federated Learning (FL) as the training and tuning protocol for applications utilizing Large Language Models (LLMs), recent research highlights the need for significant modifications to FL to accommodate the large-scale of LLMs. While substantial adjustments to the protocol have been introduced as a response, comprehensive privacy analysis for the adapted FL protocol is currently lacking. To address this gap, our work delves into an extensive examination of the privacy analysis of FL when used for training LLMs, both from theoretical and practical perspectives. In particular, we design two active membership inference attacks with guaranteed theoretical success rates to assess the privacy leakages of various adapted FL configurations. Our theoretical findings are translated into practical attacks, revealing substantial privacy vulnerabilities in popular LLMs, including BERT, RoBERTa, DistilBERT, and OpenAI's GPTs, across multiple real-world language datasets. Additionally, we conduct thorough experiments to evaluate the privacy leakage of these models when data is protected by state-of-the-art differential privacy (DP) mechanisms.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 37 of its 66 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

37 of 66 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2019Deep Leakage from Gradientscited by 1,004
25 more
  1. 2023BloombergGPT: A Large Language Model for Financecited by 923
  2. 2021Membership Inference Attacks From First Principlescited by 839
  3. 2022Galactica: A Large Language Model for Sciencecited by 736
  4. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  5. 2020Hopfield Networks is All You Needcited by 461
  6. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  8. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  9. 2014RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Responsecited by 262
  10. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  11. 2017Collecting Telemetry Data Privatelycited by 147
  12. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
  13. 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 115
  14. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 106
  15. 2021CAFE: Catastrophic Data Leakage in Vertical Federated Learningcited by 76
  16. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 49
  17. 2020Training Keyword Spotting Models on Non-IID Data with Federated Learningcited by 36
  18. 2019Federated User Representation Learningcited by 31
  19. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  20. 2023Integration of Large Language Models and Federated Learningcited by 28
  21. 2020Pretraining Federated Text Models for Next Word Predictioncited by 22
  22. 2023Active Membership Inference Attack under Local Differential Privacy in Federated Learningcited by 17
  23. 2019P3SGD: Patient Privacy Preserving SGD for Regularizing Deep CNNs in Pathological Image Classificationcited by 14
  24. 2022Multi-Freq-LDPy: Multiple Frequency Estimation Under Local Differential Privacy in Pythoncited by 5
  25. 2022Blockchain-based Secure Client Selection in Federated Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 3.

9 papers.

  1. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  2. 2026Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approachcited by 3
  3. 2025Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptationcited by 2
  4. 2024Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Securitycited by 1
  5. 2025Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Modelscited by 1
  6. 2026Burn-After-Use for Preventing Data Leakage through a Secure Multi-Tenant Architecture in Enterprise LLMcited by 1
  7. 2026Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuningcited by 1
  8. 2025A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendationcited by 0
  9. 2025Theoretically Unmasking Inference Attacks Against LDP-Protected Clients in Federated Vision Modelscited by 0