# Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research

arXiv: https://arxiv.org/abs/2602.08387 · HTML: https://mlsanity.com/paper/2602.08387/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 7 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 5 of 5)

- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training](https://mlsanity.com/paper/2410.06511/) (2024)
- [LEONARDO: A Pan-European Pre-Exascale Supercomputer for HPC and AI Applications](https://mlsanity.com/paper/2307.16885/) (2023)

## Built on by (most cited first, 2 of 2)

- [A Dual-Path Architecture for Scaling Compute and Capacity in LLMs](https://mlsanity.com/paper/2605.30202/) (2026)
- [A Sovereign, Open-Source Foundation Model for German and English](https://mlsanity.com/paper/2607.09424/) (2026)

## Abstract (arXiv)

Today's LLM (pre-) training and research workflows typically allocate a significant amount of compute to large-scale ablation studies. Despite the substantial compute costs of these ablations, existing open-source frameworks provide limited tooling for these experiments, often forcing researchers to write their own wrappers and scripts. We propose Modalities, an end-to-end PyTorch-native framework that integrates data-driven LLM research with large-scale model training from two angles. Firstly, by integrating state-of-the-art parallelization strategies, it enables both efficient pretraining and systematic ablations at trillion-token and billion-parameter scale. Secondly, Modalities adopts modular design with declarative, self-contained configuration, enabling reproducibility and extensibility levels that are difficult to achieve out-of-the-box with existing LLM training frameworks.