mlsanityData to 14 Oct 2026

AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness

arXiv 2210.07297 ↗First posted 13 Oct 2022cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Scaling up model sizes can lead to fundamentally new capabilities in many machine learning (ML) tasks. However, training big models requires strong distributed system expertise to carefully design model-parallel execution strategies that suit the model architectures and cluster setups. In this paper, we develop AMP, a framework that automatically derives such strategies. AMP identifies a valid space of model parallelism strategies and efficiently searches the space for high-performed strategies, by leveraging a cost model designed to capture the heterogeneity of the model and cluster specifications. Unlike existing methods, AMP is specifically tailored to support complex models composed of uneven layers and cluster setups with more heterogeneous accelerators and bandwidth. We evaluate AMP on popular models and cluster setups from public clouds and show that AMP returns parallel strategies that match the expert-tuned strategies on typical cluster setups. On heterogeneous clusters or models with heterogeneous architectures, AMP finds strategies with 1.54x and 1.77x higher throughput than state-of-the-art model-parallel systems, respectively.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

15 of 32 references.

References · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  8. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  9. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  10. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  11. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  12. 2017Device Placement Optimization with Reinforcement Learningcited by 86
3 more
  1. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  2. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 57
  3. 2015Distributed Balanced Partitioning via Linear Embeddingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  2. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  3. 2023Does compressing activations help model parallel training?cited by 12
  4. 2024Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Schedulingcited by 7
  5. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  6. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  7. 2024Pipette: Automatic Fine-grained Large Language Model Training Configurator for Real-World Clusterscited by 1
  8. 2026HetCCL: Accelerating LLM Training with Heterogeneous GPUscited by 1
  9. 2026AutoScout: Structured Optimization for Automating ML System Configurationcited by 1
  10. 2025Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelizationcited by 0
  11. 2025ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniquescited by 0
  12. 2026Parallelism Strategy Chaining for Fast Training Convergencecited by 0
1 more
  1. 2026HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Trainingcited by 0