mlsanityData to 14 Oct 2026

FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models

arXiv 2506.02961 ↗First posted 3 Jun 2025cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have achieved state-of-the-art results across diverse domains, yet their development remains reliant on vast amounts of publicly available data, raising concerns about data scarcity and the lack of access to domain-specific, sensitive information. Federated Learning (FL) presents a compelling framework to address these challenges by enabling decentralized fine-tuning on pre-trained LLMs without sharing raw data. However, the compatibility and performance of pre-trained LLMs in FL settings remain largely under explored. We introduce the FlowerTune LLM Leaderboard, a first-of-its-kind benchmarking suite designed to evaluate federated fine-tuning of LLMs across four diverse domains: general NLP, finance, medical, and coding. Each domain includes federated instruction-tuning datasets and domain-specific evaluation metrics. Our results, obtained through a collaborative, open-source and community-driven approach, provide the first comprehensive comparison across 26 pre-trained LLMs with different aggregation and fine-tuning strategies under federated settings, offering actionable insights into model performance, resource constraints, and domain adaptation. This work lays the foundation for developing privacy-preserving, domain-specialized LLMs for real-world applications.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 56 of its 70 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

56 of 70 references.

References · 56

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  3. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  4. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  7. 2024Phi-4 Technical Reportcited by 711
  8. 2023Instruction Tuning with GPT-4cited by 686
  9. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  10. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  11. 2024Qwen2-Audio Technical Reportcited by 560
  12. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  13. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  14. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  15. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  16. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  17. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  18. 2023FinGPT: Open-Source Financial Large Language Modelscited by 253
  19. 2024Capabilities of Gemini Models in Medicinecited by 237
  20. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  21. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  22. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  23. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  24. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  25. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  26. 2020Collaborative Fairness in Federated Learningcited by 95
  27. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  28. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 63
  29. 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 59
  30. 2021Towards Fairness-Aware Federated Learningcited by 58
  31. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
  32. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  33. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  34. 2024Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuningcited by 27
  35. 2024Automatic Instruction Evolving for Large Language Modelscited by 26
  36. 2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
  37. 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
  38. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  2. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
  3. 2026EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraintscited by 1
  4. 2025Collaborative Batch Size Optimization for Federated Learningcited by 0
  5. 2026ProToken: Token-Level Attribution for Federated Large Language Modelscited by 0
  6. 2026Knowledge-Free Correlated Agreement for Incentivizing Federated Learningcited by 0
  7. 2026Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuningcited by 0
  8. 2026Silent Failures in Federated Personalization of Foundation Modelscited by 0
  9. 2026GuidaPA: Privacy-Preserving Chatbot for Public Administration via Federated Learningcited by 0
  10. 2026TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimizationcited by 0
  11. 2026Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deploymentcited by 0