# FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health

arXiv: https://arxiv.org/abs/2509.14275 · HTML: https://mlsanity.com/paper/2509.14275/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 77 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Builds on (most cited first, 30 of 46)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases](https://mlsanity.com/paper/2402.14905/) (2024)
- [FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations](https://mlsanity.com/paper/2409.05976/) (2024)
- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [Dreaddit: A Reddit Dataset for Stress Analysis in Social Media](https://mlsanity.com/paper/1911.00133/) (2019)
- [Differentially Private Federated Learning on Heterogeneous Data](https://mlsanity.com/paper/2111.09278/) (2021)
- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [Towards Interpretable Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2304.03347/) (2023)
- [Differential Privacy-enabled Federated Learning for Sensitive Health Data](https://mlsanity.com/paper/1910.02578/) (2019)
- [Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning](https://mlsanity.com/paper/2312.17493/) (2023)
- [On Privacy and Personalization in Cross-Silo Federated Learning](https://mlsanity.com/paper/2206.07902/) (2022)
- [Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Models](https://mlsanity.com/paper/2307.11991/) (2023)
- [Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directions](https://mlsanity.com/paper/2208.03392/) (2022)
- [Fine-Tuning Large Language Models with User-Level Differential Privacy](https://mlsanity.com/paper/2407.07737/) (2024)
- [Scaling Language Model Size in Cross-Device Federated Learning](https://mlsanity.com/paper/2204.09715/) (2022)
- [Can AI Relate: Testing Large Language Model Response for Mental Health Support](https://mlsanity.com/paper/2405.12021/) (2024)
- [Do Large Language Models Align with Core Mental Health Counseling Competencies?](https://mlsanity.com/paper/2410.22446/) (2024)
- [Context-Aware Local Differential Privacy](https://mlsanity.com/paper/1911.00038/) (2019)

## Built on by (most cited first, 5 of 5)

- [Toward Zero-Egress Psychiatric AI: On-Device LLM Deployment for Privacy-Preserving Mental Health Decision Support](https://mlsanity.com/paper/2604.18302/) (2026)
- [OracleZoom: On-Policy Self-Distillation Inspired Reference-Constrained Recursive Image Super Resolution](https://mlsanity.com/paper/2609.06490/) (2026)
- [BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering](https://mlsanity.com/paper/2609.06922/) (2026)
- [FedMental: Evaluating Federated Learning for Mental Health Detection from Social Media Data](https://mlsanity.com/paper/2605.18936/) (2026)
- [New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models](https://mlsanity.com/paper/2609.11022/) (2026)

## Abstract (arXiv)

Privacy-preserving adaptation of Large Language Models (LLMs) in sensitive domains (e.g., mental health) requires balancing strict confidentiality with model utility and safety. We propose FedMentor, a federated fine-tuning framework that integrates Low-Rank Adaptation (LoRA) and domain-aware Differential Privacy (DP) to meet per-domain privacy budgets while maintaining performance. Each client (domain) applies a custom DP noise scale proportional to its data sensitivity, and the server adaptively reduces noise when utility falls below a threshold. In experiments on three mental health datasets, we show that FedMentor improves safety over standard Federated Learning (FL) without privacy, raising safe output rates by up to three points and lowering toxicity, while maintaining utility (BERTScore F1 and ROUGE-L) within 0.5% of the non-private baseline and close to the centralized upper bound. The framework scales to backbones with up to 1.7B parameters on single-GPU clients, requiring < 173 MB of communication per-round. FedMentor demonstrates a practical approach to privately fine-tune LLMs for safer deployments in healthcare and other sensitive fields.