mlsanityData to 7 Oct 2026

Pathways: Asynchronous Distributed Dataflow for ML

arXiv 2203.12533 ↗First posted 23 Mar 2022cs.DC cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

We present the design of a new large scale orchestration layer for accelerators. Our system, Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models. Pathways uses a sharded dataflow graph of asynchronous operators that consume and produce futures, and efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects. Pathways makes use of a novel asynchronous distributed dataflow design that lets the control plane execute in parallel despite dependencies in the data plane. This design, with careful engineering, allows Pathways to adopt a single-controller model that makes it easier to express complex new parallelism patterns. We demonstrate that Pathways can achieve performance parity (~100% accelerator utilization) with state-of-the-art systems when running SPMD computations over 2048 TPUs, while also delivering throughput comparable to the SPMD case for Transformer models that are pipelined across 16 stages, or sharded across two islands of accelerators connected over a data center network.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 29 of its 82 references that are in the map. First posted on arXiv 23 Mar 2022.

Builds on

29 of 82 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
17 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2017Large Batch Training of Convolutional Networkscited by 529
  3. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  4. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  5. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  6. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  7. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  8. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  9. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  10. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  11. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
  12. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
  13. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 37
  14. 2019TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learningcited by 22
  15. 2019Neural Message Passing for Multi-Label Classificationcited by 20
  16. 2020Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learningcited by 17
  17. 2018Dynamic Control Flow in Large-Scale Machine Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 23; 2023: 27; 2024: 17; 2025: 7; 2026: 7.

81 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  7. 2023Towards Generalist Biomedical AIcited by 286
  8. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  9. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  10. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  11. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  12. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
38 more
  1. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  2. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  3. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  4. 2022Generative Knowledge Graph Construction: A Reviewcited by 49
  5. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  6. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  7. 2022Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspectivecited by 40
  8. 2024Addressing cognitive bias in medical language modelscited by 34
  9. 2025Measuring the environmental impact of delivering AI at Google Scalecited by 31
  10. 2023SAFE: Machine Unlearning With Shard Graphscited by 28
  11. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 27
  12. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  13. 2023Large Language Models for User Interest Journeyscited by 24
  14. 2024Tx-LLM: A Large Language Model for Therapeuticscited by 23
  15. 2022On Optimizing the Communication of Model Parallelismcited by 19
  16. 2022OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Modelscited by 17
  17. 2022An Evolutionary Approach to Dynamic Introduction of Tasks in Large-scale Multitask Learning Systemscited by 16
  18. 2023Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capabilitycited by 16
  19. 2024Mixtures of Neural Operators Reduce Active Complexity in Operator Learningcited by 16
  20. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 15
  21. 2022XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaversecited by 14
  22. 2025Robust LLM Training Infrastructure at ByteDancecited by 14
  23. 2023Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defensecited by 13
  24. 2022Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructurescited by 12
  25. 2024Glauber Generative Model: Discrete Diffusion Models via Binary Classificationcited by 11
  26. 2024Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfacescited by 11
  27. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  28. 2022PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Modelcited by 9
  29. 2022A Continual Development Methodology for Large-scale Multitask Dynamic ML Systemscited by 9
  30. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  31. 2024DrJAX: Scalable and Differentiable MapReduce Primitives in JAXcited by 9
  32. 2023Language models are susceptible to incorrect patient self-diagnosis in medical applicationscited by 8
  33. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 8
  34. 2024Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studiescited by 7
  35. 2025Applications of Large Models in Medicinecited by 6
  36. 2022Quant 4.0: Engineering Quantitative Investment with Automated, Explainable and Knowledge-driven Artificial Intelligencecited by 5
  37. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5
  38. 2023Large Language Models and Foundation Models in Smart Agriculture: Basics, Opportunities, and Challengescited by 4

50 most cited of 81 All 81 (JSON)