# Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder

arXiv: https://arxiv.org/abs/2604.09389 · HTML: https://mlsanity.com/paper/2604.09389/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Builds on (most cited first, 8 of 8)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Scaling Laws and Interpretability of Learning from Repeated Data](https://mlsanity.com/paper/2205.10487/) (2022)
- [Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model](https://mlsanity.com/paper/2305.12816/) (2023)
- [Reducing the Transformer Architecture to a Minimum](https://mlsanity.com/paper/2410.13732/) (2024)
- [Efficient Neural Network Training via Subset Pretraining](https://mlsanity.com/paper/2410.16523/) (2024)

## Built on by (most cited first, 1 of 1)

- [Data Density Scaling Laws for Image Self-Distillation](https://mlsanity.com/paper/2610.19013/) (2026)

## Abstract (arXiv)

Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30% of the training data is sufficient to reach approximately 90% of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.