mlsanityData to 7 Oct 2026

Supporting Very Large Models using Automatic Dataflow Graph Partitioning

arXiv 1807.08887 ↗First posted 24 Jul 2018cs.DC cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

This paper presents Tofu, a system that partitions very large DNN models across multiple GPU devices to reduce per-GPU memory footprint. Tofu is designed to partition a dataflow graph of fine-grained tensor operators in order to work transparently with a general-purpose deep learning platform like MXNet. In order to automatically partition each operator, we propose to describe the semantics of an operator in a simple language which represents tensors as lambda functions mapping from tensor coordinates to values. To optimally partition different operators in a dataflow graph, Tofu uses a recursive search algorithm that minimizes the total communication cost. Our experiments on an 8-GPU machine show that Tofu enables the training of very large CNN and RNN models. It also achieves 25% - 400% speedup over alternative approaches to train very large models.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 5; 2023: 11; 2024: 10; 2025: 4; 2026: 2.

43 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  4. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  5. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  6. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  7. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  8. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  9. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  10. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  11. 2024On Scaling Up 3D Gaussian Splatting Trainingcited by 49
  12. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
31 more
  1. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  2. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  3. 2022On Optimizing the Communication of Model Parallelismcited by 19
  4. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  5. 2021OneFlow: Redesign the Distributed Deep Learning Framework from Scratchcited by 15
  6. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  7. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  8. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  9. 2021Automap: Towards Ergonomic Automated Parallelism for ML Modelscited by 12
  10. 2024GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelismcited by 9
  11. 2024Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studiescited by 7
  12. 2021Automatic Graph Partitioning for Very Large-scale Deep Learningcited by 6
  13. 2022Demand Layering for Real-Time DNN Inference with Minimized Memory Usagecited by 6
  14. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 6
  15. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5
  16. 2023Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Modelscited by 5
  17. 2024Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10cited by 5
  18. 2022Memory Safe Computations with XLA Compilercited by 3
  19. 2023TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networkscited by 3
  20. 2023Baechi: Fast Device Placement of Machine Learning Graphscited by 2
  21. 2023Auto-Parallelizing Large Models with Rhino: A Systematic Approach on Production AI Platformcited by 2
  22. 2023DISCO: Distributed Inference with Sparse Communicationscited by 2
  23. 2023UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programmingcited by 2
  24. 2023Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Searchcited by 2
  25. 2024BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Trainingcited by 2
  26. 2024LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reductioncited by 0
  27. 2024PaSE: Parallelization Strategies for Efficient DNN Trainingcited by 0
  28. 2025MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Acceleratorscited by 0
  29. 2025Tesserae: Scalable Placement Policies for Deep Learning Workloadscited by 0
  30. 2026Piper: A Programmable Distributed Training Systemcited by 0
  31. 2026Cache-Resident LLM Inference in GB-Scale Last-Level Cachescited by 0