# Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model

arXiv: https://arxiv.org/abs/2608.13277 · HTML: https://mlsanity.com/paper/2608.13277/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 10 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 9 of 9)

- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training](https://mlsanity.com/paper/2405.15319/) (2024)
- [Masked Structural Growth for 2x Faster Language Model Pre-training](https://mlsanity.com/paper/2305.02869/) (2023)
- [Deep Incubation: Training Large Models by Divide-and-Conquering](https://mlsanity.com/paper/2212.04129/) (2022)
- [DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation](https://mlsanity.com/paper/2506.14202/) (2025)
- [Model Stitching: Looking For Functional Similarity Between Representations](https://mlsanity.com/paper/2303.11277/) (2023)
- [L4: Diagnosing Large-scale LLM Training Failures via Automated Log Analysis](https://mlsanity.com/paper/2503.20263/) (2025)
- [m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers](https://mlsanity.com/paper/2402.16918/) (2024)

## Abstract (arXiv)

We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixture of Training (MoT), a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks, trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass. On a 1.3B-parameter Gemma-style model trained on C4, MoT provides a small-scale proof of mechanism: independently trained depth slices can be recomposed into a usable language model, and a quality-parity schedule reaches the same reported perplexity as the monolithic baseline. This parity setting processes more aggregate tokens and has a shorter idealized layer-equivalent critical path after aligner preparation; its effective compute advantage depends on reusing the aligner across runs. We therefore present MoT not as a general replacement for monolithic pre-training, but as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.