FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health

arXiv 2509.14275 ↗First posted 16 Sep 2025cs.CR cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Privacy-preserving adaptation of Large Language Models (LLMs) in sensitive domains (e.g., mental health) requires balancing strict confidentiality with model utility and safety. We propose FedMentor, a federated fine-tuning framework that integrates Low-Rank Adaptation (LoRA) and domain-aware Differential Privacy (DP) to meet per-domain privacy budgets while maintaining performance. Each client (domain) applies a custom DP noise scale proportional to its data sensitivity, and the server adaptively reduces noise when utility falls below a threshold. In experiments on three mental health datasets, we show that FedMentor improves safety over standard Federated Learning (FL) without privacy, raising safe output rates by up to three points and lowering toxicity, while maintaining utility (BERTScore F1 and ROUGE-L) within 0.5% of the non-private baseline and close to the centralized upper bound. The framework scales to backbones with up to 1.7B parameters on single-GPU clients, requiring < 173 MB of communication per-round. FedMentor demonstrates a practical approach to privately fine-tune LLMs for safer deployments in healthcare and other sensitive fields.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 46 of its 77 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

46 of 77 references.

References · 46

2020Language Models are Few-Shot Learnerscited by 32,419
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
2025Qwen3 Technical Reportcited by 8,820
2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
2022Large Language Models are Zero-Shot Reasonerscited by 4,374
2016Deep Learning with Differential Privacycited by 2,728
2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
2020Adaptive Federated Optimizationcited by 1,046
2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906

34 more

2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
2019Dreaddit: A Reddit Dataset for Stress Analysis in Social Mediacited by 81
2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
2023MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Modelscited by 63
2023Towards Interpretable Mental Health Analysis with Large Language Modelscited by 53
2019Differential Privacy-enabled Federated Learning for Sensitive Health Datacited by 44
2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
2023Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Modelscited by 36
2022Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directionscited by 35
2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 29
2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
2024Can AI Relate: Testing Large Language Model Response for Mental Health Supportcited by 21
2024Do Large Language Models Align with Core Mental Health Counseling Competencies?cited by 19
2019Context-Aware Local Differential Privacycited by 18
2023Challenges of Large Language Models for Mental Health Counselingcited by 17
2023An Annotated Dataset for Explainable Interpersonal Risk Factors of Mental Disturbance in Social Media Postscited by 16
2021An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregiverscited by 15
2023Towards Healthy AI: Large Language Models Need Therapists Toocited by 13
2024Federated Learning Priorities Under the European Union Artificial Intelligence Actcited by 13
2025Exploring User Security and Privacy Attitudes and Concerns Toward the Use of General-Purpose LLM Chatbots for Mental Healthcited by 13
2024Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosiscited by 12
2023FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learningcited by 10
2024WellDunn: On the Robustness and Explainability of Language Models and Large Language Models in Identifying Wellness Dimensionscited by 10
2024Cross-silo Federated Learning with Record-level Personalized Differential Privacycited by 9
2025FedMentalCare: Towards Privacy-Preserving Fine-Tuned LLMs to Analyze Mental Health Status Using Federated Learning Frameworkcited by 8
2025ProMind-LLM: Proactive Mental Health Care via Causal Reasoning with Sensor Datacited by 6
2023Protect Your Score: Contact Tracing With Differential Privacy Guaranteescited by 5
2024Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Healthcited by 5
2025Mental Health Equity in LLMs: Leveraging Multi-Hop Question Answering to Detect Amplified and Silenced Perspectivescited by 3
2022How Much User Context Do We Need? Privacy by Design in Mental Health NLP Applicationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.