# Exact-Solution Volume and Length Generalization in Transformers

arXiv: https://arxiv.org/abs/2610.07676 · HTML: https://mlsanity.com/paper/2610.07676/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 6 of 6)

- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Scalable-Softmax Is Superior for Attention](https://mlsanity.com/paper/2501.19399/) (2025)
- [Parity, Sensitivity, and Transformers](https://mlsanity.com/paper/2602.05896/) (2026)
- [Concise One-Layer Transformers Can Do Function Evaluation (Sometimes)](https://mlsanity.com/paper/2503.22076/) (2025)
- [Indexing: the Beginning and the End](https://mlsanity.com/paper/2607.22361/) (2026)
- [Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions](https://mlsanity.com/paper/2606.08768/) (2026)

## Abstract (arXiv)

Research on transformer expressivity shows whether a transformer is capable of solving a given task, but gives little indication of whether the solution, if learned, is generalizable to longer input lengths. We study this question through normalized exact-solution volume (NESV): the fraction of a bounded parameter region that achieves an exact solution on every input of length n. For fixed-width, single-layer transformers with log n-scaled attention, we establish asymptotic bounds on NESV for four tasks: FIRST (Theta(1)), MAJORITY (Theta(1/(nlog n))), INDEX (Theta(1/n3)), and PARITY (0). These results are consistent with previous empirical results: the faster the exact-solution volume decays with input length, the harder it is to length-generalize on that task. Looking deeper into INDEX, our volume analysis reveals two error sources that grow with n. Consequently, we study a transformer model that would structurally eliminate one of the terms, theoretically improving the NESV bound to Theta(n-1), and empirically achieving 85% accuracy when tested at 10× the training length, compared with the 60% accuracy of the original model. We conclude that volume analysis may be a useful approach to identify concrete sources of length sensitivity and thus provide insights into task-specific model refinements.