mlsanityData to 14 Oct 2026

Heterogeneous Parallelism for Multimodal Large Language Model Training

arXiv 2605.27678 ↗First posted 26 May 2026cs.LG cs.DCcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Foundation model training is becoming multimodal, from post-training pipelines to large-scale pretraining. As modality coverage broadens, context windows grow, and encoder LLM scales diverge, a single LLM-centric TP/CP/PP/DP/EP layout increasingly limits throughput. This coupling forces encoders to inherit LLM-driven sharding and placement choices that can add communication, limit encoder parallelism, or constrain the LLM schedule; the mismatch is most pronounced at long contexts, where LLM context parallelism is needed for the fused multimodal sequence but encoder inputs remain bounded. We present heterogeneous parallelism for multimodal large language model training, an abstraction that lets modules in one end-to-end graph use independent layouts and rank placements, supporting colocated execution on shared GPUs and non-colocated execution on disjoint rank sets. The key challenge is preserving boundary tensor semantics across independent layouts: forward activations must be materialized for the destination layout, while backward gradients must be routed back to the source layout. We address this with boundary communicators that implement forward and backward layout transforms, plus scheduling extensions for both placement modes. We evaluate optimized homogeneous, colocated heterogeneous, and non-colocated heterogeneous configurations across multimodal workloads and GPU scales to characterize when added layout and placement freedom exposes a better operating point. Across this sweep, colocated heterogeneity improves TFLOPS/GPU by up to 49.3%, while non-colocated heterogeneity improves aggregate token throughput by up to 13.0% and TFLOPS/GPU by up to 9.6%. We validate loss convergence parity against homogeneous baselines and release the system as an open-source Megatron-LM extension.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 22 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

15 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation7 Aug

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  7. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  8. 2024DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Modelscited by 16
  9. 2024GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelismcited by 9
  10. 2025DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelismcited by 7
  11. 2025Efficient Distributed MLLM Training with Cornstarchcited by 4
  12. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
2 more
  1. 2025OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Trainingcited by 3
  2. 2026Efficient Scaling of LLM Training with Flexible Context Parallelismcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs
  2. 2026BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
  3. 2025Leveraging the true depth of LLMs
  4. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  5. 2026Scaling Native Multimodal Pre-Training From Scratch
  6. 2026Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
  7. 2026NSP: Accelerating Variable-Length LLM Training via Nested Sequence Parallelism ↗
  8. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts