# Efficient Distributed MLLM Training with Cornstarch

arXiv: https://arxiv.org/abs/2503.11367 · HTML: https://mlsanity.com/paper/2503.11367/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2025.

## Built on by (most cited first, 4 of 4)

- [Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Serving](https://mlsanity.com/paper/2512.14098/) (2025)
- [Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training](https://mlsanity.com/paper/2601.17654/) (2026)
- [A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability](https://mlsanity.com/paper/2605.18750/) (2026)
- [Heterogeneous Parallelism for Multimodal Large Language Model Training](https://mlsanity.com/paper/2605.27678/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) extend the capabilities of large language models (LLMs) by combining heterogeneous model architectures to handle diverse modalities like images and audio. However, this inherent heterogeneity in MLLM model structure and data types makes makeshift extensions to existing LLM training frameworks unsuitable for efficient MLLM training. While there are a few works that have attempted to address the heterogeneity in MLLM training, their approaches are limited to only superficially considering the characteristics of MLLMs. In this paper, we present Cornstarch, an efficient distributed MLLM training framework that contemplates MLLM's unique characteristics in both model and data parallelization. Cornstarch introduces frozen-aware pipeline parallelism and token workload-balanced context parallelism to improve MLLM training throughput. Our extensive evaluation shows that Cornstarch outperforms state-of-the-art solutions by 2.26× on average in terms of MLLM training throughput. Cornstarch is an open-source project available at https://github.com/cornstarch-org/Cornstarch.