# A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets

arXiv: https://arxiv.org/abs/2505.06150 · HTML: https://mlsanity.com/paper/2505.06150/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 5 references that are in the map.
- First posted on arXiv 9 May 2025.

## Builds on (most cited first, 4 of 4)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs](https://mlsanity.com/paper/2412.13337/) (2024)
- [Reducing Biases towards Minoritized Populations in Medical Curricular Content via Artificial Intelligence for Fairer Health Outcomes](https://mlsanity.com/paper/2407.12680/) (2024)

## Built on by (most cited first, 1 of 1)

- [Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transport](https://mlsanity.com/paper/2608.11342/) (2026)

## Abstract (arXiv)

We introduce a scaling law for fine-tuning large language models (LLMs) under fixed compute budgets that explicitly accounts for data composition. Conventional approaches measure training data solely by total tokens, yet the number of examples and their average token length -- what we term dataset volume -- play a decisive role in model performance. Our formulation is tuned following established procedures. Experiments on the BRICC dataset citesalavati2024reducing and subsets of the MMLU dataset citehendrycks2021measuringmassivemultitasklanguage, evaluated under multiple subsampling strategies, reveal that data composition significantly affects token efficiency. These results motivate refined scaling laws for practical LLM fine-tuning in resource-constrained settings.