mlsanityData to 7 Oct 2026

Auto-Parallelizing Large Models with Rhino: A Systematic Approach on Production AI Platform

arXiv 2302.08141 ↗First posted 16 Feb 2023cs.DC cs.LG cs.PLcited by 2

Not on the map. Why?

Abstract (arXiv)

We present Rhino, a system for accelerating tensor programs with automatic parallelization on AI platform for real production environment. It transforms a tensor program written for a single device into an equivalent distributed program that is capable of scaling up to thousands of devices with no user configuration. Rhino firstly works on a semantically independent intermediate representation of tensor programs, which facilitates its generalization to unprecedented applications. Additionally, it implements a task-oriented controller and a distributed runtime for optimal performance. Rhino explores on a complete and systematic parallelization strategy space that comprises all the paradigms commonly employed in deep learning (DL), in addition to strided partitioning and pipeline parallelism on non-linear models. Aiming to efficiently search for a near-optimal parallel execution plan, our analysis of production clusters reveals general heuristics to speed up the strategy search. On top of it, two optimization levels are designed to offer users flexible trade-offs between the search time and strategy quality. Our experiments demonstrate that Rhino can not only re-discover the expert-crafted strategies of classic, research and production DL models, but also identify novel parallelization strategies which surpass existing systems for novel models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

32 of 51 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2016Wide Residual Networkscited by 3,057
20 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  9. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  10. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  11. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  12. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  13. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  14. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  15. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  16. 2021M6: A Chinese Multimodal Pretrainercited by 91
  17. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  18. 2020MLIR: A Compiler Infrastructure for the End of Moore's Lawcited by 64
  19. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  20. 2019Placeto: Learning Generalizable Device Placement Algorithms for Distributed Machine Learningcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performancecited by 0
  2. 2026MASS: Deep Research for Social Sciences with Memory-Augmented Social Simulationcited by 0