mlsanityData to 7 Oct 2026

Enhancing Data Quality in Federated Fine-Tuning of Foundation Models

arXiv 2403.04529 ↗First posted 7 Mar 2024cs.LG cs.AI cs.DCcited by 4

Not on the map. Why?

Abstract (arXiv)

In the current landscape of foundation model training, there is a significant reliance on public domain data, which is nearing exhaustion according to recent research. To further scale up, it is crucial to incorporate collaboration among multiple specialized and high-quality private domain data sources. However, the challenge of training models locally without sharing private data presents numerous obstacles in data quality control. To tackle this issue, we propose a data quality control pipeline for federated fine-tuning of foundation models. This pipeline computes scores reflecting the quality of training data and determines a global threshold for a unified standard, aiming for improved global performance. Our experiments show that the proposed quality control pipeline facilitates the effectiveness and reliability of the model training, leading to better performance.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

34 of 43 references.

References · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
22 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  3. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  7. 2021Model-Contrastive Federated Learningcited by 762
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  9. 2020Aligning AI With Shared Human Valuescited by 706
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  12. 2020Federated Learning with Matched Averagingcited by 564
  13. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  14. 2023Scaling Data-Constrained Language Modelscited by 391
  15. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  16. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  17. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  18. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  19. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 121
  20. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 99
  21. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
  22. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  2. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  3. 2026MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selectioncited by 3
  4. 2025CADRE: Customizable Assurance of Data Readiness in Privacy-Preserving Federated Learningcited by 0