# Selective Attention Federated Learning: Improving Privacy and Efficiency for Clinical Text Classification

arXiv: https://arxiv.org/abs/2504.11793 · HTML: https://mlsanity.com/paper/2504.11793/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 13 of 13)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare](https://mlsanity.com/paper/1907.09173/) (2019)
- [Recovering Private Text in Federated Learning of Language Models](https://mlsanity.com/paper/2205.08514/) (2022)
- [Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models](https://mlsanity.com/paper/2201.12675/) (2022)
- [StableFDG: Style and Attention Based Learning for Federated Domain Generalization](https://mlsanity.com/paper/2311.00227/) (2023)
- [Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training](https://mlsanity.com/paper/2412.04718/) (2024)

## Built on by (most cited first, 5 of 5)

- [Does Bigger Mean Better? Comparitive Analysis of CNNs and Biomedical Vision Language Modles in Medical Diagnosis](https://mlsanity.com/paper/2510.00411/) (2025)
- [SP-Mind: An Autonomous Reasoning Agent for Spatial Proteomics Analysis](https://mlsanity.com/paper/2606.24235/) (2026)
- [Toward Causal-Visual Programming: Enhancing Agentic Reasoning in Low-Code Environments](https://mlsanity.com/paper/2509.25282/) (2025)
- [Dynamic Policy Induction for Adaptive Prompt Optimization: Bridging the Efficiency-Accuracy Gap via Lightweight Reinforcement Learning](https://mlsanity.com/paper/2509.25267/) (2025)
- [Adapting Natural Language Processing Models Across Jurisdictions: A pilot Study in Canadian Cancer Registries](https://mlsanity.com/paper/2601.00787/) (2026)

## Abstract (arXiv)

Federated Learning (FL) faces major challenges regarding communication overhead and model privacy when training large language models (LLMs), especially in healthcare applications. To address these, we introduce Selective Attention Federated Learning (SAFL), a novel approach that dynamically fine-tunes only those transformer layers identified as attention-critical. By employing attention patterns to determine layer importance, SAFL significantly reduces communication bandwidth and enhances differential privacy resilience. Evaluations on clinical NLP benchmarks (i2b2 Clinical Concept Extraction and MIMIC-III discharge summaries) demonstrate that SAFL achieves competitive performance with centralized models while substantially improving communication efficiency and privacy preservation.