mlsanityData to 7 Oct 2026

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

arXiv 2304.11277 ↗First posted 21 Apr 2023cs.DC cs.AI cs.LGcited by 641

Not on the map. Why?

Cited by 71 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains. Despite the remarkable progress made in the field of machine learning systems research, which has enabled the development and exploration of large models, such abilities remain confined to a small group of advanced users and industry leaders, resulting in an implicit technical barrier for the wider community to access and leverage these technologies. In this paper, we introduce PyTorch Fully Sharded Data Parallel (FSDP) as an industry-grade solution for large model training. FSDP has been closely co-designed with several key PyTorch core components including Tensor implementation, dispatcher system, and CUDA memory caching allocator, to provide non-intrusive user experiences and high training efficiency. Additionally, FSDP natively incorporates a range of techniques and settings to optimize resource utilization across a variety of hardware configurations. The experimental results demonstrate that FSDP is capable of achieving comparable performance to Distributed Data Parallel while providing support for significantly larger models with near-linear scalability in terms of TFLOPS.

Key facts. Cited by 641 papers in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

20 of 36 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  6. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  7. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  8. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  9. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  10. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  11. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  12. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
8 more
  1. 2020Dynamic Tensor Rematerializationcited by 45
  2. 2020Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Trainingcited by 30
  3. 2020torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Modelscited by 26
  4. 2021PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformerscited by 23
  5. 2022MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloudcited by 21
  6. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  7. 2021OneFlow: Redesign the Distributed Deep Learning Framework from Scratchcited by 15
  8. 2022DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Predictioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 35; 2024: 139; 2025: 218; 2026: 249.

641 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2025Qwen-Image Technical Reportcited by 973
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  6. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  7. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  10. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  12. 2024Scaling and evaluating sparse autoencoderscited by 530
38 more
  1. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  2. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  4. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  7. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  8. 2023Challenges and Applications of Large Language Modelscited by 251
  9. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 247
  10. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  11. 2023FuXi: A cascade machine learning forecasting system for 15-day global weather forecastcited by 237
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  13. 2024Generative Representational Instruction Tuningcited by 227
  14. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  15. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  16. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  17. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  18. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  19. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  20. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  21. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  22. 2026Advancing Open-source World Modelscited by 144
  23. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 142
  24. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  25. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  26. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 123
  27. 2025Training Agents Inside of Scalable World Modelscited by 122
  28. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  29. 2024Make Your LLM Fully Utilize the Contextcited by 118
  30. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  31. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  32. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  33. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  34. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  35. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  36. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
  37. 2025Aether: Geometric-Aware Unified World Modelingcited by 94
  38. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94

50 most cited of 641 All 641 (JSON)