mlsanityData to 14 Oct 2026

Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning

arXiv 2307.09619 ↗First posted 18 Jul 2023cs.LG cs.DCcited by 29

Not on the map. Why?

Abstract (arXiv)

We introduce Dataset Grouper, a library to create large-scale group-structured (e.g., federated) datasets, enabling federated learning simulation at the scale of foundation models. This library facilitates the creation of group-structured versions of existing datasets based on user-specified partitions and directly leads to a variety of useful heterogeneous datasets that can be plugged into existing software frameworks. Dataset Grouper offers three key advantages. First, it scales to settings where even a single group's dataset is too large to fit in memory. Second, it provides flexibility, both in choosing the base (non-partitioned) dataset and in defining partitions. Finally, it is framework-agnostic. We empirically demonstrate that Dataset Grouper enables large-scale federated language modeling simulations on datasets that are orders of magnitude larger than in previous work, allowing for federated training of language models with hundreds of millions, and even billions, of parameters. Our experimental results show that algorithms like FedAvg operate more as meta-learning methods than as empirical risk minimization methods at this scale, suggesting their utility in downstream personalization and task-specific adaptation. Dataset Grouper is available at https://github.com/google-research/dataset_grouper.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 71 of its 91 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

71 of 91 references.

References · 71

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2019Advances and Open Problems in Federated Learningcited by 3,046
  12. 2016Deep Learning with Differential Privacycited by 2,728
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  8. 2020Adaptive Federated Optimizationcited by 1,046
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  10. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  13. 2018LEAF: A Benchmark for Federated Settingscited by 792
  14. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  15. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  16. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  17. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  18. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  19. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  20. 2019Fair Resource Allocation in Federated Learningcited by 394
  21. 2019Robust Aggregation for Federated Learningcited by 317
  22. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  23. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  24. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  25. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  26. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  27. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  28. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  29. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  30. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 128
  31. 2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 128
  32. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 125
  33. 2021Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learningcited by 118
  34. 2022Federated Learning with Partial Model Personalizationcited by 114
  35. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
  36. 2021Distributionally Robust Federated Averagingcited by 104
  37. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  38. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 102

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 17; 2025: 7; 2026: 1.

29 papers.

Sort:

By area:

  1. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  2. 2023User Inference Attacks on Large Language Modelscited by 36
  3. 2024Advances and Open Challenges in Federated Foundation Modelscited by 36
  4. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  5. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  6. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  7. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 29
  8. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 28
  9. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  10. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 27
  11. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  12. 2024Vertical Federated Learning for Effectiveness, Security, Applicability: A Surveycited by 14
17 more
  1. 2025POPri: Private Federated Learning using Preference-Optimized Synthetic Datacited by 14
  2. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
  3. 2025Eager Updates For Overlapped Communication and Computation in DiLoCocited by 11
  4. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9
  5. 2024DrJAX: Scalable and Differentiable MapReduce Primitives in JAXcited by 9
  6. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  7. 2025One-Shot Federated Learning with Classifier-Free Diffusion Modelscited by 7
  8. 2024Worldwide Federated Training of Language Modelscited by 5
  9. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 5
  10. 2023Leveraging Function Space Aggregation for Federated Learning at Scalecited by 4
  11. 2023Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variancecited by 2
  12. 2025Federated Adapter on Foundation Models: An Out-Of-Distribution Approachcited by 2
  13. 2024Harnessing Increased Client Participation with Cohort-Parallel Federated Learningcited by 0
  14. 2024Personalized Additive Modeling for Multi-level Federated Learningcited by 0
  15. 2024SHFL: Secure Hierarchical Federated Learning Framework for Edge Networkscited by 0
  16. 2024Photon: Federated LLM Pre-Trainingcited by 0
  17. 2026Communication-Efficient Byzantine-Robust Federated Conformal Prediction via Partial Sharingcited by 0