Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models
Not on the map. Why?
Abstract (arXiv)
Federated large language models (FedLLMs) enable cross-silo collaborative training among institutions while preserving data locality, making them appealing for privacy-sensitive domains such as law, finance, and healthcare. However, the memorization behavior of LLMs can lead to privacy risks that may cause cross-client data leakage. In this work, we study the threat of cross-client data extraction, where a semi-honest participant attempts to recover personally identifiable information (PII) memorized from other clients' data. We propose three simple yet effective extraction strategies that leverage contextual prefixes from the attacker's local data, including frequency-based prefix sampling and local fine-tuning to amplify memorization. To evaluate these attacks, we construct a Chinese legal-domain dataset with fine-grained PII annotations consistent with CPIS, GDPR, and CCPA standards, and assess extraction performance using two metrics: coverage and efficiency. Experimental results show that our methods can recover up to 56.6% of victim-exclusive PII, where names, addresses, and birthdays are particularly vulnerable. These findings highlight concrete privacy risks in FedLLMs and establish a benchmark and evaluation framework for future research on privacy-preserving federated learning. Code and data are available at https://github.com/SMILELab-FL/FedPII.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 41 of its 64 references that are in the map. First posted on arXiv 6 Jun 2025.
Builds on
41 of 64 references.
References · 41
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
- 2019Deep Leakage from Gradientscited by 1,021
- 2018How To Backdoor Federated Learningcited by 765
- 2022Quantifying Memorization Across Neural Language Modelscited by 763
- 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
- 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
29 more
- 2023Detecting Pretraining Data from Large Language Modelscited by 387
- 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
- 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
- 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
- 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 129
- 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
- 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
- 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 120
- 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 116
- 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 103
- 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
- 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
- 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 70
- 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 37
- 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
- 2023Integration of Large Language Models and Federated Learningcited by 31
- 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
- 2023Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakagecited by 28
- 2019CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehensioncited by 26
- 2025A Survey on Federated Fine-tuning of Large Language Modelscited by 24
- 2023Quantifying and Analyzing Entity-level Memorization in Large Language Modelscited by 18
- 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
- 2023FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices using a Computing Power Aware Schedulercited by 13
- 2023Source Inference Attacks: Beyond Membership Inference Attacks in Federated Learningcited by 11
- 2024Exploring Vacant Classes in Label-Skewed Federated Learningcited by 10
- 2023A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspectivecited by 9
- 2024Analysis of Privacy Leakage in Federated Large Language Modelscited by 9
- 2024Linkage on Security, Privacy and Fairness in Federated Learning: New Balances and New Perspectivescited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.