mlsanityData to 7 Oct 2026

Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning

arXiv 2310.00247 ↗First posted 30 Sep 2023cs.LG cs.DCcited by 9

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) offers privacy-preserving decentralized machine learning, optimizing models at edge clients without sharing private data. Simultaneously, foundation models (FMs) have gained traction in the artificial intelligence (AI) community due to their exceptional performance across various tasks. However, integrating FMs into FL presents challenges, primarily due to their substantial size and intensive resource requirements. This is especially true when considering the resource heterogeneity in edge FL systems. We present an adaptive framework for Resource-aware Federated Foundation Models (RaFFM) to address these challenges. RaFFM introduces specialized model compression algorithms tailored for FL scenarios, such as salient parameter prioritization and high-performance subnetwork extraction. These algorithms enable dynamic scaling of given transformer-based FMs to fit heterogeneous resource constraints at the network edge during both FL's optimization and deployment stages. Experimental results demonstrate that RaFFM shows significant superiority in resource utilization efficiency and uses fewer resources to deploy FMs to FL. Despite the lower resource consumption, target models optimized by RaFFM achieve performance on par with traditional FL methods applied to full-sized FMs. This is evident across tasks in both natural language processing and computer vision domains.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

32 of 44 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
20 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  7. 2020What is the State of Neural Network Pruning?cited by 578
  8. 2021Towards Personalized Federated Learningcited by 462
  9. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  10. 2020Adaptive Personalized Federated Learningcited by 353
  11. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  12. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  13. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  14. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  15. 2019Accelerating Federated Learning via Momentum Gradient Descentcited by 75
  16. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 49
  17. 2021Topology-Aware Network Pruning using Multi-stage Graph Embedding and Reinforcement Learningcited by 30
  18. 2020Auto Graph Encoder-Decoder for Neural Network Pruningcited by 26
  19. 2021SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learningcited by 11
  20. 2021Enabling NAS with Automated Super-Network Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 1; 2026: 3.

9 papers.

  1. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
  2. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  3. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  4. 2024FACMIC: Federated Adaptative CLIP Model for Medical Image Classificationcited by 7
  5. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  6. 2024SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learningcited by 1
  7. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  8. 2026Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuningcited by 1
  9. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1