mlsanityData to 14 Oct 2026

FDAPT: Federated Domain-adaptive Pre-training for Language Models

arXiv 2307.06933 ↗First posted 12 Jul 2023cs.LG cs.AI cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

Foundation models (FMs) have shown prominent success in a wide range of tasks. Their applicability to specific domain-task pairings relies on the availability of, both, high-quality data and significant computational resources. These challenges are not new to the field and, indeed, Federated Learning (FL) has been shown to be a promising solution in similar setups. This paper tackles the specific case of Domain-Adaptive Pre-Training (DAPT), a key step in the application of FMs. We conduct the first comprehensive empirical study to evaluate the performance of Federated Domain-Adaptive Pre-Training (FDAPT). We demonstrate that FDAPT can maintain competitive downstream task performance to the centralized baseline in both IID and non-IID situations. Finally, we propose a novel algorithm, Frozen Federated Domain-Adaptive Pre-Training (FFDAPT). FFDAPT improves the computational efficiency by 12.1% on average and exhibits similar downstream task performance to vanilla FDAPT, with general performance fluctuations remaining less than 1%.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

25 of 38 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2019Advances and Open Problems in Federated Learningcited by 3,046
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  12. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
13 more
  1. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  2. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  3. 2018Distributed learning of deep neural network over multiple agentscited by 279
  4. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  5. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  6. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
  7. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 89
  8. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  9. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  10. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 50
  11. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 23
  12. 2020Federated pretraining and fine tuning of BERT using clinical notes from multiple siloscited by 18
  13. 2021Federated Word2Vec: Leveraging Federated Learning to Encourage Collaborative Representation Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 1.

5 papers.

  1. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
  2. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  3. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  4. 2023Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptationcited by 12
  5. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1