# Empowering Federated Learning for Massive Models with NVIDIA FLARE

arXiv: https://arxiv.org/abs/2402.07792 · HTML: https://mlsanity.com/paper/2402.07792/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [Privacy-preserving Federated Brain Tumour Segmentation](https://mlsanity.com/paper/1910.00962/) (2019)
- [NVIDIA FLARE: Federated Learning from Simulation to Real-World](https://mlsanity.com/paper/2210.13291/) (2022)
- [Federated Learning for Breast Density Classification: A Real-World Implementation](https://mlsanity.com/paper/2009.01871/) (2020)
- [Fair Federated Medical Image Segmentation via Client Contribution Estimation](https://mlsanity.com/paper/2303.16520/) (2023)
- [Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentation](https://mlsanity.com/paper/2203.10144/) (2022)
- [Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentation](https://mlsanity.com/paper/2203.06338/) (2022)
- [Communication-Efficient Vertical Federated Learning with Limited Overlapping Samples](https://mlsanity.com/paper/2303.16270/) (2023)
- [ConDistFL: Conditional Distillation for Federated Learning from Partially Annotated Data](https://mlsanity.com/paper/2308.04070/) (2023)

## Built on by (most cited first, 5 of 5)

- [Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models](https://mlsanity.com/paper/2406.10630/) (2024)
- [Synergizing Foundation Models and Federated Learning: A Survey](https://mlsanity.com/paper/2406.12844/) (2024)
- [A Secure and Trustworthy Network Architecture for Federated Learning Healthcare Applications](https://mlsanity.com/paper/2404.11698/) (2024)
- [Optimizing Federated Learning in the Era of LLMs: Message Quantization and Streaming](https://mlsanity.com/paper/2511.16450/) (2025)
- [A Typed Tensor Language for Shared-State Federated Computation](https://mlsanity.com/paper/2605.21103/) (2026)

## Abstract (arXiv)

In the ever-evolving landscape of artificial intelligence (AI) and large language models (LLMs), handling and leveraging data effectively has become a critical challenge. Most state-of-the-art machine learning algorithms are data-centric. However, as the lifeblood of model performance, necessary data cannot always be centralized due to various factors such as privacy, regulation, geopolitics, copyright issues, and the sheer effort required to move vast datasets. In this paper, we explore how federated learning enabled by NVIDIA FLARE can address these challenges with easy and scalable integration capabilities, enabling parameter-efficient and full supervised fine-tuning of LLMs for natural language processing and biopharmaceutical applications to enhance their accuracy and robustness.