# Olmix: A Framework for Data Mixing Throughout LM Development

arXiv: https://arxiv.org/abs/2602.12237 · HTML: https://mlsanity.com/paper/2602.12237/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2026.

## Built on by (most cited first, 13 of 13)

- [The Design Space of Tri-Modal Masked Diffusion Models](https://mlsanity.com/paper/2602.21472/) (2026)
- [Can Generalist Agents Automate Data Curation?](https://mlsanity.com/paper/2606.04261/) (2026)
- [123D: Unifying Multi-Modal Autonomous Driving Data at Scale](https://mlsanity.com/paper/2605.08084/) (2026)
- [DataComp-VLM: Improved Open Datasets for Vision-Language Models](https://mlsanity.com/paper/2606.28551/) (2026)
- [Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization](https://mlsanity.com/paper/2603.08022/) (2026)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [Laguna M.1/XS.2 Technical Report](https://mlsanity.com/paper/2605.27605/) (2026)
- [WARP: Weight-Space Analysis for Recovering Training Data Portfolios](https://mlsanity.com/paper/2607.01686/) (2026)
- [Self-Play Pretraining with Zero Data](https://mlsanity.com/paper/2609.30063/) (2026)
- [Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model](https://mlsanity.com/paper/2603.16622/) (2026)
- [Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models](https://mlsanity.com/paper/2604.02048/) (2026)
- [Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time](https://mlsanity.com/paper/2605.15220/) (2026)
- [Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines](https://mlsanity.com/paper/2610.03087/) (2026)

## Abstract (arXiv)

Data mixing -- determining the ratios of data from different domains -- is a first-order concern for training language models (LMs). While existing mixing methods show promise, they fall short when applied during real-world LM development. We present Olmix, a framework that addresses two such challenges. First, the configuration space for developing a mixing method is not well understood -- design choices across existing methods lack justification or consensus and overlook practical issues like data constraints. We conduct a comprehensive empirical study of this space, identifying which design choices lead to a strong mixing method. Second, in practice, the domain set evolves throughout LM development as datasets are added, removed, partitioned, and revised -- a problem setting largely unaddressed by existing works, which assume fixed domains. We study how to efficiently recompute the mixture after the domain set is updated, leveraging information from past mixtures. We introduce mixture reuse, a mechanism that reuses existing ratios and recomputes ratios only for domains affected by the update. Over a sequence of five domain-set updates mirroring real-world LM development, mixture reuse matches the performance of fully recomputing the mix after each update with 74% less compute and improves over training without mixing by 11.6% on downstream tasks.